We report experiments to determine the effect of radiation damage on the phonon spectra of the most common nuclear fuel, UO$_2$. We have irradiated thin ($sim$ 300 nm) epitaxial films of UO$_2$ with 2.1 MeV He$^{2+}$ ions to 0.15 dpa and a lattice swelling of $Delta$a/a $sim$ 0.6 %, and then used grazing-incidence inelastic X-ray scattering to measure the phonon spectrum. We succeeded to observe the acoustic modes, both transverse and longitudinal, across the Brillouin zone. The phonon energies, in both the pristine and irradiated samples, are unchanged from those observed in bulk material. On the other hand, the phonon linewidths (inversely proportional to the phonon lifetimes), show a significant broadening when comparing the pristine and irradiated samples. This effect is shown to increase with phonon energy across the Brillouin zone. The decreases in the phonon lifetimes of the acoustic modes are roughly consistent with a 50 % reduction in the thermal conductivity.