Identifying the intrinsic and extrinsic origins of magneto-transport in spin-orbit coupled systems has long been a central theme in condensed matter physics. However, it has been elusive owing to the lack of an appropriate experimental tool. In this work, using terahertz time-domain spectroscopy, we unambiguously disentangle the intrinsic and extrinsic contributions to the anisotropic magnetoresistance (AMR) of a permalloy film. We find that the scattering-independent intrinsic contribution to AMR is sizable and is as large as the scattering-dependent extrinsic contribution to AMR. Moreover, the portion of intrinsic contribution to total AMR increases with increasing temperature due to the reduction of extrinsic contribution. Further investigation reveals that the reduction of extrinsic contribution is caused by the phonon/magnon-induced negative AMR. Our result will stimulate further researches on other spin-orbit-interaction-induced phenomena for which identifying the intrinsic and extrinsic contributions is important.