Motivated by recent experiments on the phonon contribution to the thermal Hall effect in the cuprates, we present an analysis of chiral phonon transport. We assume the chiral behavior arises from a non-zero phonon Hall vicosity, which is likely induced by the coupling to electrons. Phonons with a non-zero phonon Hall viscosity have an intrinsic thermal Hall conductivity, but Chen et al. (Phys. Rev. Lett. 124, 167601 (2020)) have argued that a significantly larger thermal Hall conductivity can arise from an extrinsic contribution which is inversely proportional to the density of impurities. We solve the Boltzmann equation for phonon transport and compute the temperature ($T$) dependence of the thermal Hall conductivity originating from skew scattering off point-like impurities. We find that the dominant source for thermal Hall transport is an interference between impurity skew scattering channels with opposite parity. The thermal Hall conductivity $sim T^{d+2}$ at low $T$ in $d$ dimensions, and has a window of $T$-independent behavior for $T > T_{rm imp}$, where $T_{rm imp}$ is determined by the ratio of scattering potentials with opposite parity. We also consider the role of non-specular scattering off the sample boundary, and find that it leads to negligible corrections to thermal Hall transport at low $T$.