Chiral topological semimetals are materials that break both inversion and mirror symmetries. They host interesting phenomena such as the quantized circular photogalvanic effect (CPGE) and the chiral magnetic effect. In this work, we report a comprehensive theoretical and experimental analysis of the linear and non-linear optical responses of the chiral topological semimetal RhSi, which is known to host multifold fermions. We show that the characteristic features of the optical conductivity, which display two distinct quasi-linear regimes above and below 0.4 eV, can be linked to excitations of different kinds of multifold fermions. The characteristic features of the CPGE, which displays a sign change at 0.4 eV and a large non-quantized response peak of around 160 $mu textrm{A V}^{-2}$ at 0.7 eV, are explained by assuming that the chemical potential crosses a flat hole band at the Brillouin zone center. Our theory predicts that, in order to observe a quantized CPGE in RhSi, it is necessary to increase the chemical potential as well as the quasiparticle lifetime. More broadly our methodology, especially the development of the broadband terahertz emission spectroscopy, could be widely applied to study photo-galvanic effects in noncentrosymmetric materials and in topological insulators in a contact-less way and accelerate the technological development of efficient infrared detectors based on topological semimetals.