(abridged) Far-infrared Herschel photometry from the PEP and HerMES programs is combined with ancillary datasets in the GOODS-N, GOODS-S, and COSMOS fields. Based on this rich dataset, we reproduce the restframe UV to FIR ten-colors distribution of galaxies using a superposition of multi-variate Gaussian modes. The median SED of each mode is then fitted with a modified version of the MAGPHYS code that combines stellar light, emission from dust heated by stars and a possible warm dust contribution heated by an AGN. The defined Gaussian grouping is also used to identify rare sources. The zoology of outliers includes Herschel-detected ellipticals, very blue z~1 Ly-break galaxies, quiescent spirals, and torus-dominated AGN with star formation. Out of these groups and outliers, a new template library is assembled, consisting of 32 SEDs describing the intrinsic scatter in the restframe UV-to-submm colors of infrared galaxies. This library is tested against L(IR) estimates with and without Herschel data included, and compared to eight other popular methods often adopted in the literature. When implementing Herschel photometry, these approaches produce L(IR) values consistent with each other within a median absolute deviation of 10-20%, the scatter being dominated more by fine tuning of the codes, rather than by the choice of SED templates. Finally, the library is used to classify 24 micron detected sources in PEP GOODS fields. AGN appear to be distributed in the stellar mass (M*) vs. star formation rate (SFR) space along with all other galaxies, regardless of the amount of infrared luminosity they are powering, with the tendency to lie on the high SFR side of the main sequence. The incidence of warmer star-forming sources grows for objects with higher specific star formation rates (sSFR), and they tend to populate the off-sequence region of the M*-SFR-z space.