We derive new limits on the elastic scattering cross-section between baryons and dark matter using Cosmic Microwave Background data from the Planck satellite and measurements of the Lyman-alpha forest flux power spectrum from the Sloan Digital Sky Survey. Our analysis addresses generic cross sections of the form $sigmapropto v^n$, where v is the dark matter-baryon relative velocity, allowing for constraints on the cross section independent of specific particle physics models. We include high-$ell$ polarization data from Planck in our analysis, improving over previous constraints. We apply a more careful treatment of dark matter thermal evolution than previously done, allowing us to extend our constraints down to dark matter masses of $sim$MeV. We show in this work that cosmological probes are complementary to current direct detection and astrophysical searches.