We investigate the crystal structure in multiferroic tetragonal perovskite Sr$_{1/2}$Ba$_{1/2}$MnO$_3$ with high accuracy of the order of 10$^{-3}$ Angstrom for an atomic displacement. The large atomic displacement for Mn ion from the centerosymmetric position, comparable with the off-centering distortion in the tetragonal ferroelectric BaTiO$_3$, is observed in the ferroelectric phase ($T_mathrm{N}$ $leq$ $T$ $leq$ $T_mathrm{C}$). In stark contrast, in the multiferroic phase ($T$ $leq$ $T_mathrm{N}$), the atomic displacement for Mn ion is suppressed, but those for O ions are enlarged. The atomic displacements in the polar crystal structures are also analyzed in terms of the ferroelectric modes. In the ferroelectric phase, the atomic displacements are decomposed into dominant positive Slater, negative Last, and small positive Axe modes. The suppression of Slater and Last modes, the sign change of Last mode, and the enlargement of Axe mode are found in the multiferroic phase. The ferroelectric distortion is well reproduced by a first-principles calculation based on Berry phase method, providing an additional information on competing mechanisms to induce the polarization, electronic $p$-$d$ hybridization vs. magnetic exchange-striction.