The structural and magnetic properties of the hexagonal four-layer form of SrMnO$_3$ have been investigated by combining magnetization measurements, electron diffraction and high-resolution synchrotron X-ray and neutron powder diffraction. Below 350K, there is subtle structural phase transition from hexagonal symmetry (space group $P6_3/mmc$) to orthorhombic symmetry (space group $C222_1$) where the hexagonal metric is preserved. The second-order phase transition involves a slight tilting of the corner-sharing Mn$_{2}$O$_{9}$ units composed of 2 face-sharing MnO$_6$ octahedra and the associated displacement of Sr$^{2+}$ cations. The phase transition is described in terms of symmetry-adapted displacement modes of the high symmetry phase. Upon further cooling, long range magnetic order with propagation vector $mathbf{k}=(0,0,0)$ sets in below 300K. The antiferromagnetic structure, analyzed using representation theory, shows a considerably reduced magnetic moment indicating the crucial role played by direct exchange between Mn centers of the Mn$_{2}$O$_{9}$ units.