Faint $zsim5$ quasars with $M_{1450}sim-23$ mag are known to be the potentially important contributors to the ultraviolet ionizing background in the post-reionization era. However, their number density has not been well determined, making it difficult to assess their role in the early ionization of the intergalactic medium (IGM). In this work, we present the updated results of our $zsim5$ quasar survey using the Infrared Medium-deep Survey (IMS), a near-infrared imaging survey covering an area of 85 deg$^{2}$. From our spectroscopic observations with the Gemini Multi-Object Spectrograph (GMOS) on the Gemini-South 8 m Telescope, we discovered eight new quasars at $zsim5$ with $-26.1leq M_{1450} leq -23.3$. Combining our IMS faint quasars ($M_{1450}>-27$ mag) with the brighter Sloan Digital Sky Survey (SDSS) quasars ($M_{1450}<-27$ mag), we derive the $zsim5$ quasar luminosity function (QLF) without any fixed parameters down to the magnitude limit of $M_{1450}=-23$ mag. We find that the faint-end slope of the QLF is very flat ($alpha=-1.2^{+1.4}_{-0.6}$), with a characteristic luminosity of $M^{*}_{1450}=-25.8^{+1.4}_{-1.1}$ mag. The number density of $zsim5$ quasars from the QLF gives an ionizing emissivity at 912 $unicode{x212B}$ of $epsilon_{912}=(3.7$--$7.1)times10^{23}$ erg s$^{-1}$ Hz$^{-1}$ Mpc$^{-3}$ and an ionizing photon density of $dot{n}_{rm ion}=(3.0$--$5.7)times10^{49}$ Mpc$^{-3}$ s$^{-1}$. These results imply that quasars are responsible for only 10-20% (up to 50% even in the extreme case) of the photons required to completely ionize the IGM at $zsim5$, disfavoring the idea that quasars alone could have ionized the IGM at $zsim5$.