In metals near a quantum critical point, the electrical resistance is thought to be determined by the lifetime of the carriers of current, rather than the scattering from defects. The observation of $T$-linear resistivity suggests that the lifetime only depends on temperature, implying the vanishing of an intrinsic energy scale and the presence of a quantum critical point. Our data suggest that this concept extends to the magnetic field dependence of the resistivity in the unconventional superconductor BaFe$_2$(As$_{1-x}$P$_{x}$)$_2$ near its quantum critical point. We find that the lifetime depends on magnetic field in the same way as it depends on temperature, scaled by the ratio of two fundamental constants $mu_B/k_B$. These measurements imply that high magnetic fields probe the same quantum dynamics that give rise to the $T$-linear resistivity, revealing a novel kind of magnetoresistance that does not depend on details of the Fermi surface, but rather on the balance of thermal and magnetic energy scales. This opens new opportunities for the investigation of transport near a quantum critical point by using magnetic fields to couple selectively to charge, spin and spatial anisotropies.