We numerically study the relaxation dynamics of several glass-forming models to their inherent structures, following quenches from equilibrium configurations sampled across a wide range of temperatures. In a mean-field Mari-Kurchan model, we find that relaxation changes from a power-law to an exponential decay below a well-defined temperature, consistent with recent findings in mean-field $p$-spin models. By contrast, for finite-dimensional systems, the relaxation is always algebraic, with a non-trivial universal exponent at high temperatures crossing over to a harmonic value at low temperatures. We demonstrate that this apparent evolution is controlled by a temperature-dependent population of localised excitations. Our work unifies several recent lines of studies aiming at a detailed characterization of the complex potential energy landscape of glass-formers.