The neutron-rich $^{213}$Pb isotope was produced in the fragmentation of a primary 1 GeV $A$ $^{238}$U beam, separated in FRS in mass and atomic number, and then implanted for isomer decay $gamma$-ray spectroscopy with the RISING setup at GSI. A newly observed isomer and its measured decay properties indicate that states in $^{213}$Pb are characterized by the seniority quantum number that counts the nucleons not in pairs coupled to angular momentum $J=0$. The conservation of seniority is a consequence of the Berry phase associated with particle-hole conjugation, which becomes gauge invariant and therefore observable in semi-magic nuclei where nucleons half-fill the valence shell. The $gamma$-ray spectroscopic observables in $^{213}$Pb are thus found to be driven by two mechanisms, particle-hole conjugation and seniority conservation, which are intertwined through the Berry phase.