The GRAVITY instrument on the ESO VLTI pioneers the field of high-precision near-infrared interferometry by providing astrometry at the $10 - 100,mu$as level. Measurements at such high precision crucially depend on the control of systematic effects. Here, we investigate how aberrations introduced by small optical imperfections along the path from the telescope to the detector affect the astrometry. We develop an analytical model that describes the impact of such aberrations on the measurement of complex visibilities. Our formalism accounts for pupil-plane and focal-plane aberrations, as well as for the interplay between static and turbulent aberrations, and successfully reproduces calibration measurements of a binary star. The Galactic Center observations with GRAVITY in 2017 and 2018, when both Sgr A* and the star S2 were targeted in a single fiber pointing, are affected by these aberrations at a level of less than 0.5 mas. Removal of these effects brings the measurement in harmony with the dual beam observations of 2019 and 2020, which are not affected by these aberrations. This also resolves the small systematic discrepancies between the derived distance $R_0$ to the Galactic Center reported previously.