We have performed high resolution neutron diffraction and inelastic neutron scattering experiments in the frustrated multiferroic hexagonal compounds RMnO3 (R=Ho, Yb, Sc, Y), which provide evidence of a strong magneto-elastic coupling in the the whole family. We can correlate the atomic positions, the type of magnetic structure and the nature of the spin waves whatever the R ion and temperature. The key parameter is the position of the Mn ions in the unit cell with respect to a critical threshold of 1/3, which determines the sign of the coupling between Mn triangular planes.