The decoherence induced on a single qubit by its interaction with the environment is studied. The environment is modelled as a scalar two-level boson system that can go through either first order or continuous excited state quantum phase transitions, depending on the values of the control parameters. A mean field method based on the Tamm-Damkoff approximation is worked out in order to understand the observed behaviour of the decoherence. Only the continuous excited state phase transition produces a noticeable effect in the decoherence of the qubit. This is maximal when the system-environment coupling brings the environment to the critical point for the continuous phase transition. In this situation, the decoherence factor (or the fidelity) goes to zero with a finite size scaling power law.