We performed optical spectroscopy measurement on single crystal of CeTe$_3$, a rare-earth element tri-telluride charge density wave (CDW) compound. The optical spectra are found to display very strong temperature dependence. Besides a large and pronounced CDW energy gap being present already at room temperature as observed in earlier studies, the present measurement revealed the formation of another energy gap at smaller energy scale at low temperature. The second CDW gap removes the electrons near E$_F$ which undergo stronger scattering. The study yields evidence for the presence of multiple CDW orders or strong fluctuations in the light rare-earth element tri-telluride.