Reproducing the electronic structure of AM$_4$X$_8$ lacunar spinels with a breathing pyrochlore lattice is a great theoretical challenge due to the interplay of various factors. The character of the M$_4$X$_4$ cluster orbitals is critically influenced by the Jahn-Teller instability, the spin-orbit interaction, and also by the magnetic state of the clusters. Consequently, to reproduce the narrow-gap semiconducting nature of these moderately correlated materials requires advanced approaches, since the strength of the inter-cluster hopping is strongly affected by the character of the cluster orbitals. In order to provide a solid experimental basis for theoretical studies, we performed broadband optical spectroscopy on a large set of lacunar spinels, with systematically changing ions at the A and M sites as well as the ligand (A=Ga, Ge, Al; M=V, Mo, Nb, Ta; X=S, Se). Our study covers the range of phonon excitations and also electronic transitions near the gap edge. In the phonon excitation spectrum a limited subset of the symmetry allowed modes is observed in the cubic state, with a few additional modes emerging upon the symmetry-lowering structural transition. All the infrared active modes are assigned to vibrations of the ligands and ions at the A sites, with no obvious contribution from the M-site ions. Concerning the electronic states, we found that all compounds are narrow-gap semiconductors ($E_mathrm{g} = 130 - 350,$meV) already in their room-temperature cubic state and their structural transitions induce weak, if any, changes in the band gap. The gap value is decreased when substituting S with Se and also when replacing $3d$ ions by $4d$ or $5d$ ions at the M sites.