Temperature-dependent reflectivity measurements on the kagome metal CsV$_3$Sb$_5$ in a broad frequency range of $50-20000$ cm$^{-1}$ down to $T$=10 K are reported. The charge-density wave (CDW) formed below $T_{rm CDW}$ = 94 K manifests itself in a prominent spectral-weight transfer from low to higher energy regions. The CDW gap of 60-75 meV is observed at the lowest temperature and shows significant deviations from an isotropic BCS-type mean-field behavior. Absorption peaks appear at frequencies as low as 200 cm$^{-1}$ and can be identified with interband transitions according to density-functional calculations. The change in the interband absorption compared to KV$_3$Sb$_5$ reflects the inversion of band saddle points between the K and Cs compounds. Additionally, a broader and strongly temperature-dependent absorption feature is observed below 1000 cm$^{-1}$ and assigned to a displaced Drude peak. It reflects localization effects on charge carriers.