Black phosphorus (BP) is receiving significant attention because of its direct 0.4-1.5 eV layer-dependent band gap and high mobility. Because BP devices rely on exfoliation from bulk crystals, there is a need to understand native impurities and defects in the source material. In particular, samples are typically p-doped, but the source of the doping is not well understood. Here, we use scanning tunneling microscopy and spectroscopy to compare atomic defects of BP samples from two commercial sources. Even though the sources produced crystals with an order of magnitude difference in impurity atoms, we observed a similar defect density and level of p-doping. We attribute these defects to phosphorus vacancies and provide evidence that they are the source of the p-doping. We also compare these native defects to those induced by air exposure and show they are distinct and likely more important for control of electronic structure. These results indicate that impurities in BP play a minor role compared to vacancies, which are prevalent in commercially-available materials, and call for better control of vacancy defects.