The possibility to perform high-resolution time-resolved electron energy loss spectroscopy has the potential to impact a broad range of research fields. Resolving small energy losses with ultrashort electron pulses, however, is an enormous challenge due to the low average brightness of a pulsed beam. In this letter, we propose to use time-of-flight measurements combined with longitudinal phase space manipulation using resonant microwave cavities. This allows for both an accurate detection of energy losses with a high current throughput, and efficient monochromation. First, a proof-of-principle experiment is presented, showing that with the incorporation of a compression cavity the flight time resolution can be improved significantly. Then, it is shown through simulations that by adding a cavity-based monochromation technique, a full-width-at-half-maximum energy resolution of 22 meV can be achieved with 3.1 ps pulses at a beam energy of 30 keV with currently available technology. By combining state-of-the-art energy resolutions with a pulsed electron beam, the technique proposed here opens up the way to detecting short-lived excitations within the regime of highly collective physics.