We directly measure redshift evolution in the mean physical properties (far-infrared luminosity, temperature, and mass) of the galaxies that produce the cosmic infrared background (CIB), using measurements from the Balloon-borne Large Aperture Sub-millimeter Telescope (BLAST), and Spitzer which constrain the CIB emission peak. This sample is known to produce a surface brightness in the BLAST bands consistent with the full CIB, and photometric redshifts are identified for all of the objects. We find that most of the 70 micron background is generated at z <~ 1 and the 500 micron background generated at z >~ 1. A significant growth is observed in the mean luminosity from ~ 10^9 - 10^12 L_sun, and in the mean temperature by 10 K, from redshifts 0< z < 3. However, there is only weak positive evolution in the comoving dust mass in these galaxies across the same redshift range. We also measure the evolution of the far-infrared luminosity density, and the star-formation rate history for these objects, finding good agreement with other infrared studies up to z ~1, exceeding the contribution attributed to optically-selected galaxies.