The non-relativistic hydrogen atom enjoys an accidental $SO(4)$ symmetry, that enlarges the rotational $SO(3)$ symmetry, by extending the angular momentum algebra with the Runge-Lenz vector. In the relativistic hydrogen atom the accidental symmetry is partially lifted. Due to the Johnson-Lippmann operator, which commutes with the Dirac Hamiltonian, some degeneracy remains. When the non-relativistic hydrogen atom is put in a spherical cavity of radius $R$ with perfectly reflecting Robin boundary conditions, characterized by a self-adjoint extension parameter $gamma$, in general the accidental $SO(4)$ symmetry is lifted. However, for $R = (l+1)(l+2) a$ (where $a$ is the Bohr radius and $l$ is the orbital angular momentum) some degeneracy remains when $gamma = infty$ or $gamma = frac{2}{R}$. In the relativistic case, we consider the most general spherically and parity invariant boundary condition, which is characterized by a self-adjoint extension parameter. In this case, the remnant accidental symmetry is always lifted in a finite volume. We also investigate the accidental symmetry in the context of the Pauli equation, which sheds light on the proper non-relativistic treatment including spin. In that case, again some degeneracy remains for specific values of $R$ and $gamma$.