We report new results for the elastic constants studied in Faraday and Cotton-Mouton geometry in Tb$_3$Ga$_5$O$_{12}$ (TGG), a frustrated magnetic substance with strong spin-phonon interaction and remarkable crystal-electric-field (CEF) effects. We analyze the data in the framework of CEF theory taking into account the individual surroundings of the six inequivalent Tb$^{3+}$-ion positions. This theory describes both, elastic constants in the magnetic field and as a function of temperature. Moreover we present sound-attenuation data for the acoustic Cotton-Mouton effect in TGG.