The electronic structures of the heavy-fermion superconductor CeRh$_2$As$_2$ with the local inversion symmetry breaking and the reference material LaRh$_2$As$_2$ have been investigated by using experimental optical conductivity ($sigma_1(omega)$) spectra and first-principal DFT calculations. In the low-temperature $sigma_1(omega)$ spectra of CeRh$_2$As$_2$, a $4f$-conduction electron hybridization and heavy quasiparticles are clearly indicated by a mid-infrared peak and a narrow Drude peak. In LaRh$_2$As$_2$, these features are absent in the $sigma_1(omega)$ spectrum, however, it can nicely be reproduced by DFT calculations. For both compounds, the combination between a local inversion symmetry breaking and a large spin-orbit (SO) interaction plays an important role for the electronic structure, however, the SO splitting bands could not be resolved in the $sigma_1(omega)$ spectra due to the small SO splitting size.