Perovskite photovoltaic ABX$_3$ systems are being studied due to their high energy-conversion efficiencies with current emphasis placed on pure inorganic systems. In this work, synchrotron single-crystal diffraction measurements combined with second harmonic generation measurements reveal the absence of inversion symmetry below room temperature in CsPbBr$_3$. Local structural analysis by pair distribution function and X-ray absorption fine structure methods are performed to ascertain the local ordering, atomic pair correlations, and phase evolution in a broad range of temperatures. The currently accepted space group assignments for CsPbBr$_3$ are found to be incorrect in a manner that profoundly impacts physical properties. New assignments are obtained for the bulk structure: $Im$$bar{3}$ (above $sim$ 410 K), $P$2$_1$/$m$ (between $sim$ 300 K and $sim$ 410 K), and the polar group $Pm$ (below $sim$ 300 K), respectively. The newly observed structural distortions exist in the bulk structure consistent with the expectation of previous photoluminescence and Raman measurements. High-pressure measurements reveal multiple low-pressure phases, one of which exists as a metastable phase at ambient pressure. This work should help guide research in the perovskite photovoltaic community to better control the structure under operational conditions and further improve transport and optical properties.