Gamma-ray observations have shown pulsars to be efficient converters of rotational energy into GeV photons and it is of wide-ranging interest to determine their contribution to the gamma-ray background. We arrive at flux predictions from both the young (<~ Myr) and millisecond (~Gyr) Galactic pulsar populations. We find that unresolved pulsars can yield both a significant fraction of the excess GeV gamma rays near the Galactic Center and an inverse Compton flux in the inner kpc similar to that inferred by Fermi. We compare models of the young pulsar population and millisecond pulsar population to constraints from gamma-ray and radio observations. Overall, we find that the young pulsars should outnumber millisecond pulsars as unassociated gamma-ray point sources in this region. The number of young radio pulsars discovered near the Galactic Center is in agreement with our model of the young pulsar population. Deeper radio observations at higher latitudes can constrain the total gamma-ray emission from both young and millisecond pulsars from the inner galaxy. While this is a step towards better understanding of pulsars, cosmic rays in the Milky Way, and searches for dark matter, we also discuss a few interesting puzzles that arise from the underlying physics of pulsar emission and evolution.