The nature of the pseudogap phase of cuprates remains a major puzzle. One of its new signatures is a large negative thermal Hall conductivity $kappa_{rm xy}$, which appears for dopings $p$ below the pseudogap critical doping $p^*$, but whose origin is as yet unknown. Because this large $kappa_{rm xy}$ is observed even in the undoped Mott insulator La$_2$CuO$_4$, it cannot come from charge carriers, these being localized at $p = 0$. Here we show that the thermal Hall conductivity of La$_2$CuO$_4$ is roughly isotropic, being nearly the same for heat transport parallel and normal to the CuO$_2$ planes, i.e. $kappa_{rm zy}(T) approx kappa_{rm xy} (T)$. This shows that the Hall response must come from phonons, these being the only heat carriers able to move as easily normal and parallel to the planes . At $p > p^*$, in both La$_{rm 1.6-x}$Nd$_{rm 0.4}$Sr$_x$CuO$_4$ and La$_{rm 1.8-x}$Eu$_{rm 0.2}$Sr$_x$CuO$_4$ with $p = 0.24$, we observe no c-axis Hall signal, i.e. $kappa_{rm zy}(T) = 0$, showing that phonons have zero Hall response outside the pseudogap phase. The phonon Hall response appears immediately below $p^* = 0.23$, as confirmed by the large $kappa_{rm zy}(T)$ signal we find in La$_{1.6-x}$Nd$_{rm 0.4}$Sr$_x$CuO$_4$ with $p = 0.21$. The microscopic mechanism by which phonons become chiral in cuprates remains to be identified. This mechanism must be intrinsic - from a coupling of phonons to their electronic environment - rather than extrinsic, from structural defects or impurities, as these are the same on both sides of $p^*$. This intrinsic phonon Hall effect provides a new window on quantum materials and it may explain the thermal Hall signal observed in other topologically nontrivial insulators.