We present a coherent timing analysis of the 2003 outburst of the accreting millisecond pulsar XTE J1807-294. We find an upper limit for the spin frequency derivative of 5E-14 Hz/s. The sinusoidal fractional amplitudes of the pulsations are the highest observed among the accreting millisecond pulsars and can reach values of up to 27% (2.5-30 keV). The pulse arrival time residuals of the fundamental follow a linear anti-correlation with the fractional amplitudes that suggests hot spot motion over the surface of the neutron star both in longitude and latitude. An anti-correlation between residuals and X-ray flux suggests an influence of accretion rate on pulse phase, and casts doubts on the use of standard timing techniques to measure spin frequencies and torques on the neutron star.