The Balloon-borne Large Aperture Submillimeter Telescope (BLAST) has made one square degree, deep, confusion limited maps at three different bands, centered on the Great Observatories Origins Deep Survey South field. By calculating the covariance of these maps with catalogs of 24 micron sources from the Far-Infrared Deep Extragalactic Legacy Survey (FIDEL), we have determined that the total submillimeter intensities are 8.60 +/- 0.59, 4.93 +/- 0.34, and 2.27 +/- 0.20 nW m^2 sr^(-1) at 250, 350, and 500 micron, respectively. These numbers are more precise than previous estimates of the cosmic infrared background (CIB) and are consistent with 24 micron-selected galaxies generating the full intensity of the CIB. We find that the fraction of the CIB that originates from sources at z >= 1.2 increases with wavelength, with 60% from high redshift sources at 500 micron. At all BLAST wavelengths, the relative intensity of high-z sources is higher for 24 micron-faint sources than it is for 24 micron-bright sources. Galaxies identified as active galactic nuclei (AGN) by their Infrared Array Camera (IRAC) colors are 1.6-2.6 times brighter than the average population at 250-500 micron, consistent with what is found for X-ray-selected AGN. BzK-selected galaxies are found to be moderately brighter than typical 24 micron-selected galaxies in the BLAST bands. These data provide high precision constraints for models of the evolution of the number density and intensity of star forming galaxies at high redshift.