Aims. We quantify the contributions of 24um galaxies to the Far-Infrared (FIR) Background at 70 and 160um. We provide new estimates of the Cosmic Infrared Background (CIB), and compare it with the Cosmic Optical Background (COB). Methods. Using Spitzer data at 24, 70 and 160um in three deep fields, we stacked more than 19000 MIPS 24um sources with S24>60uJy at 70 and 160um, and measured the resulting FIR flux densities. Results. This method allows a gain up to one order of magnitude in depth in the FIR. We find that the Mid-Infrared (MIR) 24um selected sources contribute to more than 70% of the CIB at 70 and 160um. This is the first direct measurement of the contribution of MIR-selected galaxies to the FIR CIB. Galaxies contributing the most to the total CIB are thus z~1 luminous infrared galaxies, which have intermediate stellar masses. We estimate that the CIB will be resolved at 0.9 mJy at 70 and 3 mJy at 160um. By combining the extrapolation of the 24um source counts below 60uJy, with 160/24 and 70/24 colors as measured with the stacking analysis, we obtain lower limits of 7.1+/-1.0 and 13.4+/-1.7 nW/m2/sr for the CIB at 70 and 160um, respectively. Conclusions. The MIPS surveys have resolved more than three quarters of the MIR and FIR CIB. By carefully integrating the Extragalactic Background Light (EBL) SED, we also find that the CIB has the same brightness as the COB, around 24 nW/m2/sr. The EBL is produced on average by 115 infrared photons for one visible photon. Finally, the galaxy formation and evolution processes emitted a brightness equivalent to 5% of the primordial electromagnetic background (CMB).