We exploit the deep and extended far infrared data sets (at 70, 100 and 160 um) of the Herschel GTO PACS Evolutionary Probe (PEP) Survey, in combination with the HERschel Multi tiered Extragalactic Survey (HerMES) data at 250, 350 and 500 um, to derive the evolution of the restframe 35 um, 60 um, 90 um, and total infrared (IR) luminosity functions (LFs) up to z~4. We detect very strong luminosity evolution for the total IR LF combined with a density evolution. In agreement with previous findings, the IR luminosity density increases steeply to z~1, then flattens between z~1 and z~3 to decrease at z greater than 3. Galaxies with different SEDs, masses and sSFRs evolve in very different ways and this large and deep statistical sample is the first one allowing us to separately study the different evolutionary behaviours of the individual IR populations contributing to the IR luminosity density. Galaxies occupying the well established SFR/stellar mass main sequence (MS) are found to dominate both the total IR LF and luminosity density at all redshifts, with the contribution from off MS sources (0.6 dex above MS) being nearly constant (~20% of the total IR luminosity density) and showing no significant signs of increase with increasing z over the whole 0.8<z<2.2 range. Sources with mass in the 10< log(M/Msun) <11 range are found to dominate the total IR LF, with more massive galaxies prevailing at the bright end of the high-z LF. A two-fold evolutionary scheme for IR galaxies is envisaged: on the one hand, a starburst-dominated phase in which the SMBH grows and is obscured by dust, is followed by an AGN dominated phase, then evolving toward a local elliptical. On the other hand, moderately starforming galaxies containing a low-luminosity AGN have various properties suggesting they are good candidates for systems in a transition phase preceding the formation of steady spiral galaxies.