From an on-going survey of the Galactic bulge, we have discovered a number of compact, steep spectrum radio sources. In this present study we have carried out more detailed observations for two of these sources, located 43 arcmin and 12.7 deg from the Galactic Center. Both sources have a very steep spectrum (alpha ~ -3) and are compact, with upper limits on the angular size of 1-2 arcsec. Their flux densities appear to be relatively steady on timescales of years, months, and hours, with no indications of rapid variability or transient behavior. We detect significant circularly polarized emission from both sources, but only weak or upper limits on linear polarization. Neither source has a counterpart at other wavelengths and deep, high-frequency searches fail to find pulsations. We compare their source properties with other known compact, non-thermal source populations in the bulge (e.g. X-ray binaries, magnetars, the Burper, cataclysmic variables). Our existing data support the hypothesis that they are scatter broadened millisecond or recycled pulsars, either at the bulge or along the line of sight. We also consider the possibility that they may be a new population of Galactic radio sources which share similar properties as pulsars but lack pulsations; a hypothesis that can be tested by future large-scale synoptic surveys.