We study the effect of oxygen vacancies on the electronic structure of the model strongly correlated metal SrVO$_3$. By means of angle-resolved photoemission (ARPES) synchrotron experiments, we investigate the systematic effect of the UV dose on the measured spectra. We observe the onset of a spurious dose-dependent prominent peak at an energy range were the lower Hubbard band has been previously reported in this compound, raising questions on its previous interpretation. By a careful analysis of the dose dependent effects we succeed in disentangling the contributions coming from the oxygen vacancy states and from the lower Hubbard band. We obtain the intrinsic ARPES spectrum for the zero-vacancy limit, where a clear signal of a lower Hubbard band remains. We support our study by means of state-of-the-art ab initio calculations that include correlation effects and the presence of oxygen vacancies. Our results underscore the relevance of potential spurious states affecting ARPES experiments in correlated metals, which are associated to the ubiquitous oxygen vacancies as extensively reported in the context of a two-dimensional electron gas (2DEG) at the surface of insulating $d^0$ transition metal oxides.