As endpoints of the hierarchical mass-assembly process, the stellar populations of local early-type galaxies encode the assembly history of galaxies over cosmic time. We use Horizon-AGN, a cosmological hydrodynamical simulation, to study the merger histories of local early-type galaxies and track how the morphological mix of their progenitors evolves over time. We provide a framework for alleviating `progenitor bias -- the bias that occurs if one uses only early-type galaxies to study the progenitor population. Early-types attain their final morphology at relatively early epochs -- by $zsim1$, around 60 per cent of todays early-types have had their last significant merger. At all redshifts, the majority of mergers have one late-type progenitor, with late-late mergers dominating at $z>1.5$ and early-early mergers becoming significant only at $z<0.5$. Progenitor bias is severe at all but the lowest redshifts -- e.g. at $zsim0.6$, less than 50 per cent of the stellar mass in todays early-types is actually in progenitors with early-type morphology, while, at $zsim2$, studying only early-types misses almost all (80 per cent) of the stellar mass that eventually ends up in local early-type systems. At high redshift, almost all massive late-type galaxies, regardless of their local environment or star-formation rate, are progenitors of local early-type galaxies, as are lower-mass (M$_star$ $<$ 10$^{10.5}$ M$_{odot}$) late-types as long as they reside in high density environments. In this new era of large observational surveys (e.g. LSST, JWST), this study provides a framework for studying how todays early-type galaxies have been built up over cosmic time.