We report a structural study of the Weyl semimetals TaAs, TaP, NbP, and NbAs, utilizing diffraction techniques (single crystal x-ray diffraction and energy dispersive spectroscopy) and imaging techniques (transmission electron microscopy/scanning transmission electron microscopy). We observe defects of various degrees, leading to non-stoichiometric single crystals of all four semimetals. While TaP displays a large pnictide deficiency with composition TaP$_{0.83(3)}$, and stacking faults accompanied by anti-site disorder and site vacancies, TaAs displays transition metal deficiency with composition Ta$_{0.92(2)}$As and a high density of stacking faults. NbP also displays pnictide deficiency, yielding composition NbP$_{0.95(2)}$, and lastly, NbAs display very little deviation from a 1:1 composition, NbAs$_{1.00(3)}$, and is therefore recommended to serve as the model compound for these semimetals.