Quantum oscillations and negative Hall and Seebeck coefficients at low temperature and high magnetic field have shown the Fermi surface of underdoped cuprates to contain a small closed electron pocket. It is thought to result from a reconstruction by charge order, but whether it is the order seen by NMR and ultrasound above a threshold field or the short-range modulations seen by X-ray diffraction in zero field is unclear. Here we use measurements of the thermal Hall conductivity in YBCO to show that Fermi-surface reconstruction occurs only above a sharply defined onset field, equal to the transition field seen in ultrasound. This reveals that electrons do not experience long-range broken translational symmetry in the zero-field ground state, and hence in zero field there is no quantum critical point for the onset of charge order as a function of doping.