We report on the optical excitation spectra in Sr$_2$VO$_4$. The phonon modes are assigned and their evolution with temperature is discussed in the frame of the different phase transitions crossed upon cooling. Besides the expected infrared-active phonons we observe two additional excitations at about 290 cm$^{-1}$ and 840 cm$^{-1}$ which could correspond to electronic transitions of the V$^{4+}$ ions. Our experimental results are discussed in the context of recent experimental and theoretical studies of this material with a unique spin-orbital ground state.