Although mergers and starbursts are often invoked in the discussion of QSO activity in the context of galaxy evolution, several studies have questioned their importance or even their presence in QSO host galaxies. Accordingly, we are conducting a study of z~0.2 QSO host galaxies previously classified as passively evolving elliptical galaxies. We present deep Keck LRIS spectroscopy of a sample of 15 hosts and model their stellar absorption spectra using stellar synthesis models. The high S/N of our spectra allow us to break various degeneracies that arise from different combinations of models, varying metallicities, and contamination from QSO light. We find that none of the host spectra can be modeled by purely old stellar populations and that the majority of the hosts (14/15) have a substantial contribution from intermediate-age populations with ages ranging from 0.7 to 2.4 Gyr. An average host spectrum is strikingly well fit by a combination of an old population and a 2.1 (+0.5, -0.7) Gyr population. The morphologies of the host galaxies suggest that these aging starbursts were induced during the early stages of the mergers that resulted in the elliptical-shaped galaxies that we observe. The current AGN activity likely corresponds to the late episodes of accretion predicted by numerical simulations, which occur near the end of the mergers, whereas earlier episodes may be more difficult to observe due to obscuration. Our off-axis observations prevent us from detecting any current star formation or young stellar populations that may be present in the central few kiloparsecs.