Gauge invariance is the cornerstone of modern quantum field theory. Recently, it has been shown that the quantum Rabi model, describing the dipolar coupling between a two-level atom and a quantized electromagnetic field, violates this principle. This widely used model describes a plethora of quantum systems and physical processes under different interaction regimes. In the ultrastrong coupling regime, it provides predictions which drastically depend on the chosen gauge. This failure is attributed to the finite-level truncation of the matter system. We show that a careful application of the gauge principle is able to restore gauge invariance even for extreme light-matter interaction regimes. The resulting quantum Rabi Hamiltonian in the Coulomb gauge differs significantly from the standard model and provides the same physical results obtained by using the dipole gauge. It contains field operators to all orders that cannot be neglected when the coupling strength is high. These results shed light on subtleties of gauge invariance in nonperturbative and extreme interaction regimes, which are now experimentally accessible, and solve all the long-lasting controversies arising from gauge ambiguities in the quantum Rabi and Dicke models.