We measure the temperature and frequency dependence of the complex Hall angle for normal state YBa$_2$Cu$_3$O$_7$ films from dc to far-infrared frequencies (20-250 cm$^{-1}$) using a new modulated polarization technique. We determine that the functional dependence of the Hall angle on scattering does not fit the expected Lorentzian response. We find spectral evidence supporting models of the Hall effect where the scattering $Gamma_H$ is linear in T, suggesting that a single relaxation rate, linear in temperature, governs transport in the cuprates.