We recently proposed a method to constrain $s$-wave annihilating MeV dark matter from a combination of the Voyager 1 and the AMS-02 data on cosmic-ray electrons and positrons. Voyager 1 actually provides an unprecedented probe of dark matter annihilation to cosmic rays down to $sim 10$ MeV in an energy range where the signal is mostly immune to uncertainties in cosmic-ray propagation. In this article, we derive for the first time new constraints on $p$-wave annihilation down to the MeV mass range using cosmic-ray data. To proceed, we derive a self-consistent velocity distribution for the dark matter across the Milky Way by means of the Eddington inversion technique and its extension to anisotropic systems. As inputs, we consider state-of-the-art Galactic mass models including baryons and constrained on recent kinematic data. They allow for both a cored or a cuspy halo. We then calculate the flux of cosmic-ray electrons and positrons induced by $p$-wave annihilating dark matter and obtain very stringent limits in the MeV mass range, robustly excluding cross sections greater than $sim 10^{-22}{rm cm^3/s}$ (including theoretical uncertainties), about 5 orders of magnitude better than current CMB constraints. This limit assumes that dark matter annihilation is the sole source of cosmic rays and could therefore be made even more stringent when reliable models of astrophysical backgrounds are included.