The precise determination of the position of point-like emitters and scatterers using far-field optical imaging techniques is of utmost importance for a wide range of applications in medicine, biology, astronomy, and physics. Although the optical wavelength sets a fundamental limit to the image resolution of unknown objects, the position of an individual emitter can in principle be estimated from the image with arbitrary precision. This is used, e.g., in stars position determination and in optical super-resolution microscopy. Furthermore, precise position determination is an experimental prerequisite for the manipulation and measurement of individual quantum systems, such as atoms, ions, and solid state-based quantum emitters. Here we demonstrate that spin-orbit coupling of light in the emission of elliptically polarized emitters can lead to systematic, wavelength-scale errors in the estimate of the emitters position. Imaging a single trapped atom as well as a single sub-wavelength-diameter gold nanoparticle, we demonstrate a shift between the emitters measured and actual positions which is comparable to the optical wavelength. Remarkably, for certain settings, the expected shift can become arbitrarily large. Beyond their relevance for optical imaging techniques, our findings apply to the localization of objects using any type of wave that carries orbital angular momentum relative to the emitters position with a component orthogonal to the direction of observation.