We use a simple optical/infrared (IR) photometric selection of high-redshift QSOs that identifies a Lyman Break in the optical photometry and requires a red IR color to distinguish QSOs from common interlopers. The search yields 100 z~3 (U-dropout) QSO candidates with 19<r<22 over 11.7 deg^2 in the ELAIS-N1 (EN1) and ELAIS-N2 (EN2) fields of the Spitzer Wide-area Infrared Extragalactic (SWIRE) Legacy Survey. The z~3 selection is reliable, with spectroscopic follow-up of 10 candidates confirming they are all QSOs at 2.83<z<3.44. We find that our z~4$ (g-dropout) sample suffers from both unreliability and incompleteness but present 7 previously unidentified QSOs at 3.50<z<3.89. Detailed simulations show our z~3 completeness to be ~80-90% from 3.0<z<3.5, significantly better than the ~30-80% completeness of the SDSS at these redshifts. The resulting luminosity function extends two magnitudes fainter than SDSS and has a faint end slope of beta=-1.42 +- 0.15, consistent with values measured at lower redshift. Therefore, we see no evidence for evolution of the faint end slope of the QSO luminosity function. Including the SDSS QSO sample, we have now directly measured the space density of QSOs responsible for ~70% of the QSO UV luminosity density at z~3. We derive a maximum rate of HI photoionization from QSOs at z~3.2, Gamma = 4.8x10^-13 s^-1, about half of the total rate inferred through studies of the Ly-alpha forest. Therefore, star-forming galaxies and QSOs must contribute comparably to the photoionization of HI in the intergalactic medium at z~3.