The intergalactic medium (IGM) at $zsim$ 5 to 6 is largely ionized, and yet the main source for the IGM ionization in the early universe is uncertain. Of the possible contributors are faint quasars with $-26 lesssim M_{rm 1450} lesssim -23$, but their number density is poorly constrained at $zsim5$. In this paper, we present our survey of faint quasars at $zsim5$ in the European Large-Area {it ISO} Survey-North 1 (ELAIS-N1) field over a survey area of 6.51 deg$^2$ and examine if such quasars can be the dominant source of the IGM ionization. We use the deep optical/near-infrared data of the ELAIS-N1 field as well as the additional medium-band observations to find $z sim 5$ quasars through a two-step approach using the broadband color selection, and SED fitting with the medium-band information included. Adopting Bayesian information criterion, we identify ten promising quasar candidates. Spectra of three of the candidates are obtained, confirming all of them to be quasars at $zsim5$ and supporting the reliability of the quasar selection. Using the promising candidates, we derive the $zsim5$ quasar luminosity function at $-26 lesssim M_{rm 1450} lesssim -23$. The number density of faint $zsim5$ quasars in the ELAIS-N1 field is consistent with several previous results that quasars are not the main contributors to the IGM-ionizing photons at $zsim5$