The ultrafast response of metals to light is governed by intriguing non-equilibrium dynamics involving the interplay of excited electrons and phonons. The coupling between them gives rise to nonlinear diffusion behavior on ultrashort timescales. Here, we use scanning ultrafast thermo-modulation microscopy to image the spatio-temporal hot-electron diffusion in a thin gold film. By tracking local transient reflectivity with 20 nm and 0.25 ps resolution, we reveal two distinct diffusion regimes, consisting of an initial rapid diffusion during the first few picoseconds after optical excitation, followed by about 100-fold slower diffusion at longer times. We simulate the thermo-optical response of the gold film with a comprehensive three-dimensional model, and identify the two regimes as hot-electron and phonon-limited thermal diffusion, respectively.