The nature of the atomic defects on the hydrogen passivated Si (100) surface is analyzed using deep learning and scanning tunneling microscopy (STM). A robust deep learning framework capable of identifying atomic species, defects, in the presence of non-resolved contaminates, step edges, and noise is developed. The automated workflow, based on the combination of several networks for image assessment, atom-finding and defect finding, is developed to perform the analysis at different levels of description and is deployed on an operational STM platform. This is further extended to unsupervised classification of the extracted defects using the mean-shift clustering algorithm, which utilizes features automatically engineered from the combined output of neural networks. This combined approach allows the identification of localized and extended defects on the topographically non-uniform surfaces or real materials. Our approach is universal in nature and can be applied to other surfaces for building comprehensive libraries of atomic defects in quantum materials.