We searched for X-ray serendipitous sources in over 370 Swift-XRT fields centered on gamma ray bursts detected between 2004 and 2008 and observed with total exposures ranging from 10 ks to over 1 Ms. This defines the Swift Serendipitous Survey in deep XRT GRB fields, which is quite broad compared to existing surveys (~33 square degrees) and medium depth, with a faintest flux limit of 7.2e-16 erg cm^-2 s^-1 in the 0.5 to 2 keV energy range. The survey has a high degree of uniformity thanks to the stable point spread function and small vignetting correction factors of the XRT, moreover is completely random on the sky as GRBs explode in totally unrelated parts of the sky. In this paper we present the sample and the X-ray number counts of the high Galactic-latitude sample, estimated with high statistics over a wide flux range (i.e., 7.2e-16 to ~5e-13 erg cm^-2 s^-1 in the 0.5-2 keV band and 3.4e-15 to ~6e-13 erg cm^-2 s^-1 in the 2-10 keV band). We detect 9387 point-like sources, while 7071 point-like sources are found at high Galactic-latitudes (i.e. >=20 deg). The large number of detected sources resulting from the combination of large area and deep flux limits make this survey a new important tool for investigating the evolution of AGN. In particular, the large area permits finding rare high-luminosity objects like QSO2, which are poorly sampled by other surveys, adding precious information for the luminosity function bright end. The high Galactic-latitude logN-logS relation is well determined over all the flux coverage, and it is nicely consistent with previous results at 1 sigma confidence level. By the hard X-ray color analysis, we find that the Swift Serendipitous Survey in deep XRT GRB fields samples relatively unobscured and mildly obscured AGN, with a fraction of obscured sources of ~37% (~15%) in the 2-10 (0.3-3 keV) band.