Motivated by experimental observations, Samajdar et al. [Nature Physics 15, 1290 (2019)] have proposed that the insulating Neel state in the parent compounds of the cuprates is proximate to a quantum phase transition to a state in which Neel order coexists with semion topological order. We study the manner in which proximity to this transition can make the phonons chiral, by inducing a significant phonon Hall viscosity. We describe the spinon-phonon coupling in a lattice spinon model coupled to a strain field, and also using a general continuum theory constrained only by symmetry. We find a nonanalytic Hall viscosity across the transition, with a divergent second derivative at zero temperature.