The delafossite metals PdCoO$_{2}$, PtCoO$_{2}$ and PdCrO$_{2}$ are among the highest conductivity materials known, with low temperature mean free paths of tens of microns in the best as-grown single crystals. A key question is whether these very low resistive scattering rates result from strongly suppressed backscattering due to special features of the electronic structure, or are a consequence of highly unusual levels of crystalline perfection. We report the results of experiments in which high energy electron irradiation was used to introduce point disorder to the Pd and Pt layers in which the conduction occurs. We obtain the cross-section for formation of Frenkel pairs in absolute units, and cross-check our analysis with first principles calculations of the relevant atomic displacement energies. We observe an increase of resistivity that is linear in defect density with a slope consistent with scattering in the unitary limit. Our results enable us to deduce that the as-grown crystals contain extremely low levels of in-plane defects of approximately $0.001%$. This confirms that crystalline perfection is the most important factor in realizing the long mean free paths, and highlights how unusual these delafossite metals are in comparison with the vast majority of other multi-component oxides and alloys. We discuss the implications of our findings for future materials research.