When noninteracting fermions are confined in a $D$-dimensional region of volume $mathrm{O}(L^D)$ and subjected to a continuous (or piecewise continuous) potential $V$ which decays sufficiently fast with distance, in the thermodynamic limit, the ground state energy of the system does not depend on $V$. Here, we discuss this theorem from several perspectives and derive a proof for radially symmetric potentials valid in $D$ dimensions. We find that this universality property holds under a quite mild condition on $V$, with or without bounded states, and extends to thermal states. Moreover, it leads to an interesting analogy between Andersons orthogonality catastrophe and first-order quantum phase transitions.