Temperature dependent crystal structures of the quasi-one-dimensional ladder material BaFe2Se3 are examined. Combining the optical second harmonic generation (SHG) experiments and neutron diffraction measurements, we elucidate the crystal structure with Pmn2_1 space group in the low-temperature phase below Ts2 = 400 K, further above Neel temperature. This low-temperature phase loses the spatial inversion symmetry, where a resultant macroscopic polarization emerges along the rung direction. The transition is characterized by block-type lattice distortions with the magneto-striction mechanism. Change in the electrical resistivity and the magnetic susceptibility across the polar-nonpolar transition also suggests a modification of the electronic states reflecting the structural instability. Consistency and discrepancy with the existing theory are discussed.