The optical conductivity spectra of the rattling phonons in the clathrate Ba$_8$Ga$_{16}$Ge$_{30}$ are investigated in detail by use of the terahertz time-domain spectroscopy. The experiment has revealed that the lowest-lying vibrational mode of a Ba
(2)$^{2+}$ ion consists of a sharp Lorentzian peak at 1.2 THz superimposed on a broad tail weighted in the lower frequency regime around 1.0 THz. With decreasing temperature, an unexpected linewidth broadening of the phonon peak is observed, together with monotonic softening of the phonon peak and the enhancement of the tail structure. These observed anomalies are discussed in terms of impurity scattering effects on the hybridized phonon system of rattling and acoustic phonons.
