There is now a large consensus that the current epoch of the Cosmic Star Formation History (CSFH) is dominated by low mass galaxies while the most active phase at 1<z<2 is dominated by more massive galaxies, which undergo a faster evolution. Massive galaxies tend to inhabit very massive halos such as galaxy groups and clusters. We aim to understand whether the observed galaxy downsizing could be interpreted as a halo downsizing, whereas the most massive halos, and their galaxy populations, evolve more rapidly than the halos of lower mass. Thus, we study the contribution to the CSFH of galaxies inhabiting group-sized halos. This is done through the study of the evolution of the Infra-Red (IR) luminosity function of group galaxies from redshift 0 to ~1.6. We use a sample of 39 X-ray selected groups in the Extended Chandra Deep Field South (ECDFS), the Chandra Deep Field North (CDFN), and the COSMOS field, where the deepest available mid- and far-IR surveys have been conducted with Spitzer MIPS and Hersche PACS. Groups at low redshift lack the brightest, rarest, and most star forming IR-emitting galaxies observed in the field. Their IR-emitting galaxies contribute <10% of the comoving volume density of the whole IR galaxy population in the local Universe. At redshift >~1, the most IR-luminous galaxies (LIRGs and ULIRGs) are preferentially located in groups, and this is consistent with a reversal of the star-formation rate vs .density anti-correlation observed in the nearby Universe. At these redshifts, group galaxies contribute 60-80% of the CSFH, i.e. much more than at lower redshifts. Below z~1, the comoving number and SFR densities of IR-emitting galaxies in groups decline significantly faster than those of all IR-emitting galaxies. Our results are consistent with a halo downsizing scenario and highlight the significant role of environment quenching in shaping the CSFH.