Close to a zero temperature transition between ordered and disordered electronic phases, quantum fluctuations can lead to a strong enhancement of the electron mass and to the emergence of competing phases such as superconductivity. A correlation between the existence of such a quantum phase transition and superconductivity is quite well established in some heavy fermion and iron-based superconductors and there have been suggestions that high temperature superconductivity in the copper oxide materials (cuprates) may also be driven by the same mechanism. Close to optimal doping, where the superconducting transition temperature $T_c$ is maximum in the cuprates, two different phases are known to compete with superconductivity: a poorly understood pseudogap phase and a charge ordered phase. Recent experiments have shown a strong increase in quasiparticle mass $m^*$ in the cuprate YBa$_2$Cu$_3$O$_{7-delta}$ as optimal doping is approached suggesting that quantum fluctuations of the charge ordered phase may be responsible for the high-$T_c$ superconductivity. We have tested the robustness of this correlation between $m^*$ and $T_c$ by performing quantum oscillation studies on the stoichiometric compound YBa$_2$Cu$_4$O$_8$ under hydrostatic pressure. In contrast to the results for YBa$_2$Cu$_3$O$_{7-delta}$, we find that in YBa$_2$Cu$_4$O$_8$ the mass decreases as $T_c$ increases under pressure. This inverse correlation between $m^*$ and $T_c$ suggests that quantum fluctuations of the charge order enhance $m^*$ but do not enhance $T_c$.