Electrons/positrons produced in a pulsar magnetosphere emit synchrotron radiation, which is widely believed as the origin of the non-thermal X-ray emission detected from pulsars. Particles are produced by curvature photons emitted from accelerated particles in the magnetosphere. These curvature photons are detected as pulsed $gamma$-ray emissions from pulsars with age $lesssim10^6$ yr. Using $gamma$-ray observations and analytical model, we impose severe constraints on the synchrotron radiation as a mechanism of the non-thermal X-ray emission. In most middle-aged pulsars ($sim10^5-10^6$ yr) which photon-photon pair production is less efficient in their magnetosphere, we find that the synchrotron radiation model is difficult to explain the observed non-thermal X-ray emission.