We develop a gauge theory of the critical behavior of the topological excitations-driven Berezinskii-Kosterlitz-Thouless (BKT) phase transition in the XY model with weak quenched disorder. We find that while in two-dimensions the liquid of topological defects exhibits the BKT critical behavior, the three-dimensional system shows more singular Vogel-Fulcher-Tamman criticality heralding its freezing into a spin glass. Our findings provide insights into the topological origin of spin glass formation.