The highly mobile electrons at the interface of SrTiO3 with other oxide insulators, such as LaAlO3 or AlOx, are of great current interest. A vertical gate voltage allows controlling a metal/superconductor-to-insulator transition, as well as electrical modulation of the spin-orbit Rashba coupling for spin-charge conversion. These findings raise important questions about the origin of the confined electrons as well as the mechanisms that govern the interfacial electric field. Here we use infrared ellipsometry and confocal Raman spectroscopy to show that an anomalous polar moment is induced at the interface that is non-collinear, highly asymmetric and hysteretic with respect to the vertical gate electric field. Our data indicate that an important role is played by the electromigration of oxygen vacancies and their clustering at the antiferrodistortive domain boundaries of SrTiO3, which generates local electric and possibly also flexoelectric fields and subsequent polar moments with a large lateral component. Our results open new perspectives for the defect engineering of lateral devices with strongly enhanced and hysteretic local electric fields that can be manipulated with various other parameters, like strain, temperature, or photons.