The interplay between charge density wave (CDW) order and superconductivity has attracted much attention. This is the central issue of along standing debate in simple transition metal dichalcogenides without strong electronic correlations, such as 2H-NbSe$_2$, in which twosuch phases coexist. The importance of anisotropic electron-phonon interaction has been recently highlighted from both theoretical and experimental point of view, and explains some of the key features of the formation of the CDW in this system. On the other hand, other aspects, such as the effects of anharmonicity, remain poorly understood despite their manifest importance in such soft-phonon driven phase transition. At the theoretical level in particular, their prohibitive computational price usually prevents their investigation within conventional perturbative approaches.Here, we address this issue using a combination of high resolution inelastic X-ray scattering measurements of the phonon dispersion, as afunction of temperature and pressure, with state of the art ab initio calculations. By explicitly taking into account anharmonic effects, we obtain an accurate, quantitative, description of the (P,T) dependence of the phonon spectrum, accounting for the rapid destruction of the CDW under pressure by zero mode vibrations - or quantum fluctuations - of the lattice. The low-energy longitudinal acoustic mode that drives the CDW transition barely contributes to superconductivity, explaining the insensitivity of the superconducting critical temperature to the CDW transition.