We study the dynamics of a spin ensemble strongly coupled to a single-mode resonator driven by external pulses. When the mean frequency of the spin ensemble is in resonance with the cavity mode, damped Rabi oscillations are found between the spin ensemble and the cavity mode which we describe very accurately, including the dephasing effect of the inhomogeneous spin broadening. We demonstrate that a precise knowledge of this broadening is crucial both for a qualitative and a quantitative understanding of the temporal spin-cavity dynamics. On this basis we show that coherent oscillations between the spin ensemble and the cavity can be enhanced by a few orders of magnitude, when driving the system with pulses that match special resonance conditions. Our theoretical approach is tested successfully with an experiment based on an ensemble of negatively charged nitrogen-vacancy (NV) centers in diamond strongly coupled to a superconducting coplanar single-mode waveguide resonator.