Strong interaction between light and matter waves, such as electron beams in electron microscopes, has recently emerged as a new tool for understanding entanglement. Here, we systematically investigate electron-light interactions from first principles. We show that enhanced coupling can be achieved for systems involving slow electron wavepackets interacting with plasmonic nanoparticles, due to simultaneous classical recoil and quantum mechanical photon absorption and emission processes. For slow electrons with longitudinal broadenings longer than the dimensions of nanoparticles, phase-matching between slow electrons and plasmonic oscillations is manifested as an additional degree of freedom to control the strength of coupling. Our findings pave the way towards a systematic and realistic understanding of electron-light interactions beyond adiabatic approximations, and lay the ground for realization of entangled electron-photon systems and Boson-sampling devices involving light and matter waves.