We demonstrate that the onset of complex spin orders in ACr$_2$O$_4$ spinels with magnetic A$=$Co, Fe and Cu ions lowers the lattice symmetry. This is clearly indicated by the emergence of anisotropic lattice dynamics -- as evidenced by the pronounce
d phonon splittings -- even when experiments probing static distortions fail. We show that the crystal symmetry in the magnetic phase is reduced from tetragonal to orthorhombic for FeCr$_2$O$_4$ and CuCr$_2$O$_4$ with Jahn-Teller active A-site ions. The conical spin structure in FeCr$_2$O$_4$ is also manifested in the phonon frequencies. In contrast, the multiferroic CoCr$_2$O$_4$ with no orbital degrees of freedom remains nearly cubic in its ground state.
