We investigate theoretically the effect of a magnetic field on intersubband polaritons in an asymmetric quantum well placed inside an optical resonator. It is demonstrated that the field-induced diamagnetic shift of electron subbands in the well increases the broadening of optical lines corresponding to intersubband electron transitions. As a consequence, the magnetic field can switch the polariton system from the regime of strong light-matter coupling to the regime of weak one. This effect paves a way to the effective control of polaritonic devices with a magnetic field.