We have previously calculated the intergalactic background light (IBL) as a function of redshift in the far ultraviolet to near infrared range, based purely on data from deep galaxy surveys. Here we utilize similar methods to determine the mid- and far infrared IBL out to a wavelength of 850 microns. Our approach enables us to constrain the range of photon densities, based on the uncertainties from observationally determined luminosity densities and colors. By also including the effect of the 2.7 K cosmic background photons, we determine 68% confidence upper and lower limits on the opacity of the universe to gamma-rays up to PeV energies. Our direct results on the IBL are consistent with those from complimentary gamma-ray analyses using observations from the Fermi $gamma$-ray space telescope and the H.E.S.S. air Cherenkov telescope. Thus, we find no evidence of previously suggested processes for the modification of gamma-ray spectra other than that of absorption by pair production alone.