The quantum coherence and gate fidelity of electron spin qubits in semiconductors is often limited by noise arising from coupling to a bath of nuclear spins. Isotopic enrichment of spin-zero nuclei such as $^{28}$Si has led to spectacular improvements of the dephasing time $T_2^*$ which, surprisingly, can extend two orders of magnitude beyond theoretical expectations. Using a single-atom $^{31}$P qubit in enriched $^{28}$Si, we show that the abnormally long $T_2^*$ is due to the controllable freezing of the dynamics of the residual $^{29}$Si nuclei close to the donor. Our conclusions are supported by a nearly parameter-free modeling of the $^{29}$Si nuclear spin dynamics, which reveals the degree of back-action provided by the electron spin as it interacts with the nuclear bath. This study clarifies the limits of ergodic assumptions in analyzing many-body spin-problems under conditions of strong, frequent measurement, and provides novel strategies for maximizing coherence and gate fidelity of spin qubits in semiconductors.