Dark matter interactions with electrons or protons during the early Universe leave imprints on the cosmic microwave background and the matter power spectrum, and can be probed through cosmological and astrophysical observations. We explore these interactions using a diverse suite of data: cosmic microwave background anisotropies, baryon acoustic oscillations, the Lyman-$alpha$ forest, and the abundance of Milky-Way subhalos. We derive constraints using model-independent parameterizations of the dark matter--electron and dark matter--proton interaction cross sections and map these constraints onto concrete dark matter models. Our constraints are complementary to other probes of dark matter interactions with ordinary matter, such as direct detection, big bang nucleosynthesis, various astrophysical systems, and accelerator-based experiments.