Longitudinal relaxation is the process by which an excited spin ensemble decays into its thermal equilibrium with the environment. In solid-state spin systems relaxation into the phonon bath usually dominates over the coupling to the electromagnetic vacuum. In the quantum limit the spin lifetime is determined by phononic vacuum fluctuations. However, this limit was not observed in previous studies due to thermal phonon contributions or phonon-bottleneck processes. Here we use a dispersive detection scheme based on cavity quantum electrodynamics (cQED) to observe this quantum limit of spin relaxation of the negatively charged nitrogen vacancy ($mathrm{NV}^-$) centre in diamond. Diamond possesses high thermal conductivity even at low temperatures, which eliminates phonon-bottleneck processes. We observe exceptionally long longitudinal relaxation times $T_1$ of up to 8h. To understand the fundamental mechanism of spin-phonon coupling in this system we develop a theoretical model and calculate the relaxation time ab initio. The calculations confirm that the low phononic density of states at the $mathrm{NV}^-$ transition frequency enables the spin polarization to survive over macroscopic timescales.