We perform a systematic analysis of models with GeV-scale dark matter coupled to baryons and leptons. Such theories provide a natural framework to explain the matter-antimatter asymmetry of the universe. We find that only a few baryonic dark matter models are free from tree-level proton decay without explicitly imposing baryon number conservation. We enumerate those cases and provide a brief overview of their phenomenology. We then focus on a leptonic dark matter model for a more detailed discussion of the baryon asymmetry generation via leptogenesis, the symmetry restoration in the dark sector and the expected dark matter annihilation signals in indirect detection experiments.