We report a study on the thermal conductivity of CuFe$_{1-x}$Ga$_x$O$_2$ ($x =$ 0--0.12) single crystals at temperatures down to 0.3 K and in magnetic fields up to 14 T. CuFeO$_2$ is a well-known geometrically frustrated triangular lattice antiferromagnet and can be made to display multiferroicity either by applying magnetic field along the $c$ axis or by doping nonmagnetic impurities, accompanied with rich behaviors of magnetic phase transitions. The main experimental findings of this work are: (i) the thermal conductivities ($kappa_a$ and $kappa_c$) show drastic anomalies at temperature- or field-induced magnetic transitions; (ii) the low-$T$ $kappa(H)$ isotherms exhibit irreversibility in a broad region of magnetic fields; (iii) there are phonon scattering effect caused by magnetic fluctuations at very low temperatures. These results demonstrate strong spin-phonon coupling in this material and reveal the non-negligible magnetic fluctuations in the ground state of pure and Ga-doped samples.