We study astrometric residuals from a simultaneous fit of Hyper Suprime-Cam images. We aim to characterize these residuals and study the extent to which they are dominated by atmospheric contributions for bright sources. We use Gaussian process interpolation, with a correlation function (kernel), measured from the data, to smooth and correct the observed astrometric residual field. We find that Gaussian process interpolation with a von Karman kernel allows us to reduce the covariances of astrometric residuals for nearby sources by about one order of magnitude, from 30 mas$^2$ to 3 mas$^2$ at angular scales of ~1 arcmin, and to halve the r.m.s. residuals. Those reductions using Gaussian process interpolation are similar to recent result published with the Dark Energy Survey dataset. We are then able to detect the small static astrometric residuals due to the Hyper Suprime-Cam sensors effects. We discuss how the Gaussian process interpolation of astrometric residuals impacts galaxy shape measurements, in particular in the context of cosmic shear analyses at the Rubin Observatory Legacy Survey of Space and Time.