Most old distant radio galaxies should be extended X-ray sources due to inverse Compton scattering of Cosmic Microwave Background (CMB) photons. Such sources can be an important component in X-ray surveys for high redshift clusters, due to the increase with redshift of both the CMB energy density and the radio source number density. We estimate a lower limit to the space density of such sources and show that inverse Compton scattered emission may dominate above redshifts of one and X-ray luminosities of 10^44 erg/s, with a space density of radio galaxies > 10^-8 Mpc^-3. The X-ray sources may last longer than the radio emission and so need not be associated with what is seen to be a currently active radio galaxy.