Surface photovoltage (SPV) spectroscopy, which is a versatile method to analyze the energetic distribution of electronic defect states at surfaces and interfaces of wide-bandgap semiconductor (hetero-)structures, is applied to comparatively investigate heterostructures made of 5-unit-cell-thick LaAlO$_3$ films grown either on TiO$_2$- or on SrO-terminated SrTiO$_3$. As shown in a number of experimental and theoretical investigations in the past, these two interfaces exhibit dramatically different properties with the first being conducting and the second insulating. Our present SPV investigation reveals clearly distinguishable interface defect state distributions for both configurations when interpreted within the framework of a classical semiconductor band scheme. Furthermore, bare SrTiO$_3$ crystals with TiO$_2$ or mixed SrO/TiO$_2$ terminations show similar SPV spectra and transients as do LaAlO$_3$-covered samples with the respective termination of the SrTiO$_3$ substrate. This is in accordance with a number of recent works that stress the decisive role of SrTiO$_3$ and the minor role of LaAlO$_3$ with respect to the electronic interface properties.