The electronic, optical, and magnetic properties of quantum solids are determined by their low-energy (< 100 meV) many-body excitations. Dynamical characterization and manipulation of such excitations relies on tools that combine nm-spatial, fs-temporal, and meV-spectral resolution. Currently, phonons and collective plasmon resonances can be imaged in nanostructures with sub-nm and 10s meV space/energy resolution using state-of-the-art energy-filtered transmission electron microscopy (TEM), but only under static conditions, while fs-resolved measurements are common but lack spatial or energy resolution. Here, we demonstrate a new method of spectrally resolved photon-induced near-field electron microscopy (SRPINEM) that allows us to obtain nm-fs-resolved maps of nanoparticle plasmons with an energy resolution determined by the laser linewidth (20 meV in this work), and not limited by electron beam and spectrometer energy spreading. This technique can be extended to any optically-accessible low-energy mode, thus pushing TEM to a previously inaccessible spectral domain with an unprecedented combination of space, energy and temporal resolution.