The Fermi-LAT Galactic Center excess and the 511 keV positron-annihilation signal from the inner Galaxy bare a striking morphological similarity. We propose that both can be explained through a scenario in which millisecond pulsars produce the Galactic Center excess and their progenitors, low-mass X-ray binaries, the 511 keV signal. As a proof-of-principle we study a specific population synthesis scenario from the literature involving so-called ultracompact X-ray binaries. Moreover, for the first time, we quantitatively show that neutron star, rather than black hole, low-mass X-ray binaries can be responsible for the majority of the positrons. In this particular scenario binary millisecond pulsars can be both the source of the Fermi-LAT $gamma$-ray excess and the bulge positrons. Future avenues to test this scenario are discussed.