We present proper motion measurements for more than $0.55$ million main-sequence stars, by comparing astrometric positions of matched stars between the multi-band imaging datasets from the Hyper Suprime-Cam (HSC) Survey and the SDSS Stripe 82. In doing this we use $3$ million galaxies to recalibrate the astrometry and set up a common reference frame between the two catalogues. The exquisite depth and the nearly $12$ years of time baseline between HSC and SDSS enable high-precision measurements of statistical proper motions for stars down to $isimeq 24$. A validation of our method is demonstrated by the agreement with the $Gaia$ proper motions, to the precision better than $0.1$ mas yr$^{-1}$. To retain the precision, we make a correction of the subtle effects due to the differential chromatic refraction in the SDSS images based on the comparison with the $Gaia$ proper motions against colour of stars, which is validated using the SDSS spectroscopic quasars. Combining with the photometric distance estimates for individual stars based on the precise HSC photometry, we show a significant detection of the net proper motions for stars in each bin of distance out to $100$ kpc. The two-component tangential velocities after subtracting the apparent motions due to our own motion display rich phase-space structures including a clear signature of the Sagittarius stream in the halo region of distance range $[10, 35]$ kpc. We also measure the tangential velocity dispersion in the distance range $5-20$ kpc and find that the data are consistent with a constant isotropic dispersion of $80pm 10 {rm km/s}$. More distant stars appear to have random motions with respect to the Galactic centre on average.