We study a system of a single qubit (or a few qubits) interacting with a soft-mode bosonic field. Considering an extended version of the Rabi model with both parity-conserving and parity-violating interactions, we disclose a complex arrangement of quantum phase transitions in the ground- and excited-state domains. An experimentally testable signature of some of these transitions is a dynamical stabilization of a fully factorized qubit-field state involving the field vacuum. It happens in the ultrastrong coupling regime where the superradiant field equilibrium is far from the vacuum state. The degree of stabilization varies abruptly with interaction parameters and increases with the softness of the field mode. We analyze semiclassical origins of these effects and show their connection to various forms of excited-state quantum phase transitions.