The phase diagram of cuprate high-temperature superconductors features an enigmatic pseudogap region that is characterized by a partial suppression of low energy electronic excitations. Polarized neutron diffraction, Nernst effect, THz polarimetery and ultrasound measurements on YBa$_2$Cu$_3$O$_y$ suggest that the pseudogap onset below a temperature T* coincides with a bona fide thermodynamic phase transition that breaks time-reversal, four-fold rotation and mirror symmetries respectively. However, the full point group above and below T* has not been resolved and the fate of this transition as T* approaches the superconducting critical temperature T$_c$ is poorly understood. Here we reveal the point group of YBa$_2$Cu$_3$O$_y$ inside its pseudogap and neighboring regions using high sensitivity linear and second harmonic optical anisotropy measurements. We show that spatial inversion and two-fold rotational symmetries are broken below T* while mirror symmetries perpendicular to the Cu-O plane are absent at all temperatures. This transition occurs over a wide doping range and persists inside the superconducting dome, with no detectable coupling to either charge ordering or superconductivity. These results suggest that the pseudogap region coincides with an odd-parity order that does not arise from a competing Fermi surface instability and exhibits a quantum phase transition inside the superconducting dome.