We have measured the contribution of submillimeter and mid-infrared sources to the extragalactic background radiation at 70 and 160um. Specifically, we have stacked flux in 70 and 160um Spitzer Space Telescope (Spitzer) observations of the Canada-UK Deep Sub-millimeter Survey 14h field at the positions of 850um sources detected by SCUBA and also 8 and 24um sources detected by Spitzer. We find that per source, the SCUBA galaxies are the strongest and the 8um sources the weakest contributors to the background flux at both 70 and 160um. Our estimate of the contribution of the SCUBA sources is higher than previous estimates. However, expressed as a total contribution, the full 8um source catalogue accounts for twice the total 24um source contribution and ~10 times the total SCUBA source contribution. The 8um sources account for the majority of the background radiation at 160um with a flux of 0.87+/-0.16 MJy/sr and at least a third at 70um with a flux of 0.103+/-0.019 MJy/sr. These measurements are consistent with current lower limits on the background at 70 and 160um. Finally, we have investigated the 70 and 160um emission from the 8 and 24um sources as a function of redshift. We find that the average 70um flux per 24um source and the average 160um flux per 8 and 24um source is constant over all redshifts, up to z~4. In contrast, the low-redshift half (z<1) of the of 8um sample contributes approximately four times the total 70um flux of the high-redshift half. These trends can be explained by a single non-evolving SED.