Magnetism in the insulating BaFe$_2$Se$_3$ was examined through susceptibility, specific heat, resistivity and neutron diffraction measurements. After formation of a short-range magnetic correlation, a long-range ordering was observed below $T_{rm N}
sim 255$ K. The transition is obscured by bulk properties. Magnetic moments ($parallel a$) are arranged to form a Fe$_4$ ferromagnetic unit, and each Fe$_4$ stacks antiferromagnetically. This block magnetism is of the third type among magnetic structures of ferrous materials. The magnetic ordering drives unusually large distortion via magnetoelastic coupling.
