In this paper we present a coherent timing analysis of the 401 Hz pulsations of the accreting millisecond X-ray pulsar SAX J1808.4-3658 during its 2019 outburst. Using observations collected with the Neutron Star Interior Composition Explorer (NICER), we establish the pulsar spin frequency and orbital phase during its latest epoch. We find that the 2019 outburst shows a pronounced evolution in pulse phase over the course of the outburst. These phase shifts are found to correlate with the source flux, and are interpreted in terms of hot-spot drift on the stellar surface, driven by changes in the mass accretion rate. Additionally, we find that the long-term evolution of the pulsar spin frequency shows evidence for a modulation at the Earths orbital period, enabling pulsar timing based astrometry of this accreting millisecond pulsar.