Within the complex phase diagram of the hole-doped cuprates, seizing the nature of the mysterious pseudo-gap phase is essential to unravel the microscopic origin of high-temperature superconductivity. Below the pseudo-gap temperature $rm T^{star}$, evidences for intra-unit-cell orders breaking the 4-fold rotation symmetry have been provided by neutron diffraction and scanning tunneling spectroscopy. Using polarized neutron diffraction on a detwinned $rm YBa_2Cu_3O_{6.6}$ sample, we here report a distinct a-b anisotropy of the intra-unit-cell magnetic structure factor below $rm T^{star}$, highlighting that intra-unit-cell order in this material breaks the mirror symmetry of the CuO$_2$ bilayers. This is likely to originate from a crisscrossed arrangement of loop currents within the $rm CuO_2$ bilayer, resulting in a bilayer mean toroidal axis along the $rm {bf b}$ direction.