The accreting millisecond pulsar SAX J1808.4-3658 has shown a peculiar orbital evolution in the past with an orbital expansion much faster than expected from standard binary evolutionary scenarios. Previous limits on the pulsar spin frequency derivative during transient accretion outbursts were smaller than predicted by standard magnetic accretion torque theory, while the spin evolution between outbursts was consistent with magnetic dipole spin-down. In this paper we present the results of a coherent timing analysis of the 2011 outburst observed by the Rossi X-ray Timing Explorer and extend our previous long-term measurements of the orbital and spin evolution over a baseline of thirteen years. We find that the expansion of the 2 hr orbit is accelerating at a rate 1.6E-20 s/s^2 and we interpret this as the effect of short-term angular momentum exchange between the mass donor and the orbit. The gravitational quadrupole coupling due to variations in the oblateness of the companion can be a viable mechanism for explaining the observations. No significant spin frequency derivatives are detected during the 2011 outburst (<4E-13 Hz/s) and the long term spin down remains stable over thirteen years with a rate of approximately -1E-15 Hz/s.