We report on the effects of hydrostatic pressure (HP) on the charge density wave observed in underdoped cuprates. We studied YBa$_2$Cu$_3$O$_{6.6}$ ($T_c$=61 K) using high-resolution inelastic x-ray scattering (IXS), and reveal an extreme sensitivity of the phonon anomalies related to the charge density wave (CDW) order to HP. The amplitudes of the normal state broadening and superconductivity induced phonon softening at Q$_{CDW}$ rapidly decrease as HP is applied, resulting in the complete suppression of signatures of the CDW below $sim$1 GPa. Additional IXS measurements on YBa$_2$Cu$_3$O$_{6.75}$ demonstrate that this very rapid effect cannot be explained by pressure-induced modification of the doping level and highlight the different role of external pressure and doping in tuning the phase diagram of the cuprates. Our results provide new insights into the mechanisms underlying the CDW formation and its interplay with superconductivity.