The Cosmic Far-Infrared Background (CIB) at wavelengths around 160 {mu}m corresponds to the peak intensity of the whole Extragalactic Background Light, which is being measured with increasing accuracy. However, the build up of the CIB emission as a function of redshift, is still not well known. Our goal is to measure the CIB history at 70 {mu}m and 160 {mu}m at different redshifts, and provide constraints for infrared galaxy evolution models. We use complete deep Spitzer 24 {mu}m catalogs down to about 80 {mu}Jy, with spectroscopic and photometric redshifts identifications, from the GOODS and COSMOS deep infrared surveys covering 2 square degrees total. After cleaning the Spitzer/MIPS 70 {mu}m and 160 {mu}m maps from detected sources, we stacked the far-IR images at the positions of the 24 {mu}m sources in different redshift bins. We measured the contribution of each stacked source to the total 70 and 160 {mu}m light, and compare with model predictions and recent far-IR measurements made with Herschel/PACS on smaller fields. We have detected components of the 70 and 160 {mu}m backgrounds in different redshift bins up to z ~ 2. The contribution to the CIB is maximum at 0.3 <= z <= 0.9 at 160{mu}m (and z <= 0.5 at 70 {mu}m). A total of 81% (74%) of the 70 (160) {mu}m background was emitted at z < 1. We estimate that the AGN relative contribution to the far-IR CIB is less than about 10% at z < 1.5. We provide a comprehensive view of the CIB buildup at 24, 70, 100, 160 {mu}m. IR galaxy models predicting a major contribution to the CIB at z < 1 are in agreement with our measurements, while our results discard other models that predict a peak of the background at higher redshifts. Our results are available online http://www.ias.u-psud.fr/irgalaxies/ .