Orbital monitoring of exoplanetary and stellar systems is fundamental for analysing their architecture, dynamical stability and evolution, and mechanisms of formation. Current high-contrast extreme-adaptive optics imagers like SPHERE, GPI, and SCExAO+CHARIS explore the population of giant exoplanets and brown dwarf and stellar companions beyond typically 10 au, covering generally a small fraction of the orbit (<20%) leading to degeneracies and biases in the orbital parameters. Precise and robust measurements over time of the position of the companions are critical, which require good knowledge of the instrumental limitations and dedicated observing strategies. The homogeneous dedicated calibration strategy for astrometry implemented for SPHERE has facilitated high-precision studies by its users since its start of operation in 2014. As the precision of exoplanet imaging instruments is now reaching milliarcseconds and is expected to improve with the upcoming facilities, we initiated a community effort, triggered by the SPHERE experience, to share lessons learned for high-precision astrometry in direct imaging. A homogeneous strategy would strongly benefit the VLT community, in synergy with VLTI instruments like GRAVITY/GRAVITY+, future instruments like ERIS and MAVIS, and in preparation for the exploitation of the ELTs first instruments MICADO, HARMONI, and METIS.