Co$_3$O$_4$, ZnFe$_2$O$_4$, CoFe$_2$O$_4$, ZnCo$_2$O$_4$, and Fe$_3$O$_4$ thin films were fabricated by pulsed laser deposition at high and low temperatures resulting in crystalline single-phase normal, inverse, as well as disordered spinel oxide thin films with smooth surface morphology. The dielectric function, determined by spectroscopic ellipsometry in a wide spectral range from 0.5 eV to 8.5 eV, is compared with the magneto-optical response of the dielectric tensor, investigated by magneto-optical Kerr effect (MOKE) spectroscopy in the spectral range from 1.7 eV to 5.5 eV with an applied magnetic field of 1.7 T. Crystal field, inter-valence and inter-sublattice charge transfer transitions, and transitions from O$_{2p}$ to metal cation 3d or 4s bands are identified in both the principal diagonal elements and the magneto-optically active off-diagonal elements of the dielectric tensor. Depending on the degree of cation disorder, resulting in local symmetry distortion, the magneto-optical response is found to be strongest for high crystal quality inverse spinels and for disordered normal spinel structure, contrary to the first principle studies of CoFe$_2$O$_4$ and ZnFe$_2$O$_4$. The results presented provide a basis for deeper understanding of light-matter interaction in this material system that is of vital importance for device-related phenomena and engineering.