Recently, halide perovskites have gained significant attention from the perspective of efficient spintronics owing to Rashba effect. This effect occurs as a consequence of strong spin-orbit coupling under noncentrosymmetric environment, which can be dynamic and/or static. However, there exist intense debates on the origin of broken inversion symmetry since the halide perovskites typically crystallize into a centrosymmetric structure. In order to clarify the issue, we examine both dynamic and static effects in the all-inorganic CsPbBr3 and organic-inorganic CH3NH3PbBr3 (MAPbBr3) perovskite single crystals by employing temperature- and polarization-dependent photoluminescence excitation spectroscopy. The perovskite single crystals manifest the dynamic effect by photon recycling in the indirect Rashba gap, causing dual peaks in the photoluminescence. But the effect vanishes in CsPbBr3 at low temperatures (< 50 K), accompanied by a striking color change of the crystal, arising presumably from lower degrees of freedom for inversion symmetry breaking associated with the thermal motion of the spherical Cs cation, compared with the polar MA cation in MAPbBr3. We also show that static Rashba effect occurs only in MAPbBr3 below 90 K due to surface reconstruction via MA-cation ordering, which likely extends across a few layers from the crystal surface to the interior. We further demonstrate that this static Rashba effect can be completely suppressed upon surface treatment with poly methyl methacrylate (PMMA) coating. We believe that our results provide a rationale for the Rashba effects in halide perovskites.