Recently, the High Altitude Water Cherenkov (HAWC) collaboration reported the discovery of the TeV halo around the Geminga pulsar. The TeV emission is believed to originate from inverse Compton scattering of pulsar-injected electrons/positrons off cosmic microwave background photons. In the mean time, these electrons should inevitably radiate X-ray photons via the synchrotron radiation, providing a useful constraint on the magnetic field in the TeV halo. In this work, we analyse the data of XMM-Newton and Chandra, and obtain an upper limit for the diffuse X-ray flux in a region of $600$ around the Geminga pulsar, which is at a level of $lesssim 10^{-14}rm erg,cm^{-2}s^{-1}$. Through a numerical modelling on both the X-ray and the TeV observations assuming isotropic diffusion of injected electrons/positrons, we find the magnetic field inside the TeV halo is required to be $<1mu$G, which is significantly weaker than the typical magnetic field in the interstellar medium. The weak magnetic field together with the small diffusion coefficient inferred from HAWCs observation implies that the Bohm limit of particle diffusion may probably have been achieved in the TeV halo. We also discuss alternative possibilities for the weak X-ray emission, such as the hadronic origin of the TeV emission or a specific magnetic field topology, in which a weak magnetic field and a very small diffusion coefficient might be avoided.