How did the dwarf galaxy population of present-day galaxy clusters form and grow over time? We address this question by analysing the history of dark matter subhaloes in the Millennium-II cosmological simulation. A semi-analytic model serves as the link to observations. We argue that a reasonable analogue to early morphological types or red-sequence dwarf galaxies are those subhaloes that experienced strong mass loss, or alternatively those that have spent a long time in massive haloes. This approach reproduces well the observed morphology-distance relation of dwarf galaxies in the Virgo and Coma clusters, and thus provides insight into their history. Over their lifetime, present-day late types have experienced an amount of environmental influence similar to what the progenitors of dwarf ellipticals had already experienced at redshifts above two. Therefore, dwarf ellipticals are more likely to be a result of early and continuous environmental influence in group- and cluster-size haloes, rather than a recent transformation product. The observed morphological sequences of late-type and early-type galaxies have developed in parallel, not consecutively. Consequently, the characteristics of todays late-type galaxies are not necessarily representative for the progenitors of todays dwarf ellipticals. Studies aiming to reproduce the present-day dwarf population thus need to start at early epochs, model the influence of various environments, and also take into account the evolution of the environments themselves.