We report on optical spectroscopy on the lacunar spinels GaV$_4$S$_8$ and GeV$_4$S$_8$ in the spectral range from 100 to 23000 cm$^{-1}$ and for temperatures from 5 to 300 K. These multiferroic spinel systems reveal Jahn-Teller driven ferroelectricity and complex magnetic order at low temperatures. We study the infrared-active phonon modes and the low-lying electronic excitations in the cubic high-temperature phase, as well as in the orbitally and in the magnetically ordered low-temperature phases. We compare the phonon modes in these two compounds, which undergo different symmetry-lowering Jahn-Teller transitions into ferroelectric and orbitally ordered phases, and exhibit different magnetic ground states. We follow the splitting of the phonon modes at the structural phase transition and detect additional splittings at the onset of antiferromagnetic order in GeV$_4$S$_8$. We observe electronic transitions within the $d$-derived bands of the V$_4$ clusters and document a significant influence of the structural and magnetic phase transitions on the narrow electronic band gaps.