We present a flexible and efficient ultrafast time-resolved spontaneous Raman spectroscopy setup to study collective excitation and quasi-particle dynamics in quantum matter. The setup has a broad energy tuning range extending from the visible to near infrared spectral regions for both the pump excitation and Raman probe pulses. Additionally, the balance between energy and time-resolution can be controlled. A high light collecting efficiency is realized by high numerical aperture collection optics and a high-throughput flexible spectrometer. We demonstrate the functionality of the setup with a study of the zone-center longitudinal optical phonon and hole continuum dynamics in silicon, and discuss the role of the Raman tensor in time-resolved Raman scattering. In addition, we show evidence for unequal phonon softening rates at different high symmetry points in the Brillouin zone of silicon by means of detecting pump-induced changes in the two-phonon overtone spectrum. Demagnetization dynamics in the helimagnet Cu2OSeO3 is studied by observing softening and broadening of a magnon after photo-excitation, underlining the unique power of measuring transient dynamics in the frequency domain, and the feasibility to study phase transitions in quantum matter.