We theoretically investigate the quasiparticle scattering rate $varGamma$ inside a vortex core in the existence of non-magnetic impurities distributed randomly in a superconductor. We show that the dependence of $varGamma$ on the magnetic field direction is sensitive to the sign of the pair potential. The behavior of $varGamma$ is quite different between an s-wave and a d-wave pair potential, where these are assumed to have the same amplitude anisotropy, but a sign change only for the d-wave one. It is suggested that measurements of the microwave surface impedance with changing applied-field directions would be used for the phase-sensitive identification of pairing symmetry.