We present a systematic study of the magnetic properties of semiconducting ZnFe$_2$O$_4$ thin films fabricated by pulsed laser deposition at low and high oxygen partial pressure and annealed in oxygen and argon atmosphere, respectively. The magnetic response is enhanced by annealing the films at 250$^{circ}$C and diminished at annealing temperatures above 300$^{circ}$C. The initial increase is attributed to the formation of oxygen vacancies after argon treatment, evident by the increase in the low energy absorption at $sim$ 0.9 eV involving Fe$^{2+}$ cations. The weakened magnetic response is related to a decline in disorder with a cation redistribution toward a normal spinel configuration. The structural renormalization is consistent with the decrease and increase in oscillator strength of respective electronic transitions involving tetrahedrally (at $sim$ 3.5 eV) and octahedrally (at $sim$ 5.7 eV) coordinated Fe$^{3+}$ cations.