We report a structural study of the Weyl semimetals TaAs and TaP, utilizing diffraction and imaging techniques, where we show that they contain a high density of defects, leading to non-stoichiometric single crystals of both semimetals. Despite the observed defects and non-stoichiometry on samples grown using techniques already reported in the literature, de Haas-van Alphen measurements on TaP reveal quantum oscillations and a high carrier mobility, an indication that the crystals are of quality comparable to those reported elsewhere. Electronic structure calculations on TaAs reveal that the position of the Weyl points relative to the Fermi level shift with the introduction of vacancies and stacking faults. In the case of vacancies the Fermi surface becomes considerably altered, while the effect of stacking faults on the electronic structure is to allow the Weyl pockets to remain close to the Fermi surface. The observation of quantum oscillations in a non-stoichiometric crystal and the persistence of Weyl fermion pockets near the Fermi surface in a crystal with stacking faults point to the robustness of these quantum phenomena in these materials.