We present a systematic study of the crystal field interactions in the Li$R$F$_4$, $R$ = Gd, Ho, Er, Tm and Yb, family of rare-earth magnets. Using detailed inelastic neutron scattering measurements we have been able to quantify the transition energies and wavefunctions for each system. This allows us to quantitatively describe the high-temperature susceptibility measurements for the series of materials and make predictions based on a mean-field approach for the low-temperature thermal and quantum phase transitions. We show that coupling between crystal field and phonon states leads to lineshape broadening in LiTmF$_4$ and level splitting in LiYbF$_4$. Furthermore, using high resolution neutron scattering from LiHoF$_4$, we find anomalous broadening of crystal-field excitations which we attribute to magnetoelastic coupling.