The mass distribution of compact objects provides a fossil record that can be studied to uncover information on the late stages of massive star evolution, the supernova explosion mechanism, and the dense matter equation of state. Observations of neutron star masses indicate a bimodal Gaussian distribution, while the observed black hole mass distribution decays exponentially for stellar-mass black holes. We use these observed distributions to directly confront the predictions of stellar evolution models and the neutrino-driven supernova simulations of Sukhbold et al. (2016). We find excellent agreement between the black hole and low-mass neutron star distributions created by these simulations and the observations. We show that a large fraction of the stellar envelope must be ejected, either during the formation of stellar-mass black holes or prior to the implosion through tidal stripping due to a binary companion, in order to reproduce the observed black hole mass distribution. We also determine the origins of the bimodal peaks of the neutron star mass distribution, finding that the low-mass peak (centered at ~1.4 M_sun) originates from progenitors with M_zams ~ 9-18 M_sun. The simulations fail to reproduce the observed peak of high-mass neutron stars (centered at ~1.8 M_sun) and we explore several possible explanations. We argue that the close agreement between the observed and predicted black hole and low-mass neutron star mass distributions provides new promising evidence that these stellar evolution and explosion models are accurately capturing the relevant stellar, nuclear, and explosion physics involved in the formation of compact objects.