Two-dimensional van der Waals MnBi$_{2n}$Te$_{3n+1}$ (n = 1, 2, 3, 4) compounds have been recently found to be intrinsic magnetic topological insulators rendering quantum anomalous Hall effect and diverse topological states. Here, we summarize and compare the crystal and magnetic structures of this family, and discuss the effects of chemical composition on their magnetism. We found that a considerable fraction of Bi occupies at the Mn sites in MnBi$_{2n}$Te$_{3n+1}$ (n = 1, 2, 3, 4) while Mn is no detectable at the non-magnetic atomic sites within the resolution of neutron diffraction experiments. The occupancy of Mn monotonically decreases with the increase of n. The polarized neutron diffraction on the representative MnBi$_{4}$Te$_{7}$ reveals that its magnetization density is exclusively accumulated at the Mn site, in good agreement with the results from the unpolarized neutron diffraction. The defects of Bi at the Mn site naturally explain the continuously reduced saturated magnetic moments from n = 1 to n = 4. The experimentally estimated critical exponents of all the compounds generally suggest a three-dimensional character of magnetism. Our work provides material-specified structural parameters that may be useful for band structure calculations to understand the observed topological surface states and for designing quantum magnetic materials through chemical doping.