At $z=1-3$, the formation of new stars is dominated by dusty galaxies whose far-IR emission indicates they contain colder dust than local galaxies of a similar luminosity. We explore the reasons for the evolving IR emission of similar galaxies over cosmic time using: 1) Local galaxies from GOALS $(L_{rm IR}=10^{11}-10^{12},L_odot)$; 2) Galaxies at $zsim0.1-0.5$ from the 5MUSES ($L_{rm IR}=10^{10}-10^{12},L_odot$); 3) IR luminous galaxies spanning $z=0.5-3$ from GOODS and Spitzer xFLS ($L_{rm IR}>10^{11},L_odot$). All samples have Spitzer mid-IR spectra, and Herschel and ground-based submillimeter imaging covering the full IR spectral energy distribution, allowing us to robustly measure $L_{rm IR}^{rmscriptscriptstyle SF}$, $T_{rm dust}$, and $M_{rm dust}$ for every galaxy. Despite similar infrared luminosities, $z>0.5$ dusty star forming galaxies have a factor of 5 higher dust masses and 5K colder temperatures. The increase in dust mass is linked with an increase in the gas fractions with redshift, and we do not observe a similar increase in stellar mass or star formation efficiency. $L_{160}^{rmscriptscriptstyle SF}/L_{70}^{rmscriptscriptstyle SF}$, a proxy for $T_{rm dust}$, is strongly correlated with $L_{rm IR}^{rmscriptscriptstyle SF}/M_{rm dust}$ independently of redshift. We measure merger classification and galaxy size for a subsample, and there is no obvious correlation between these parameters and $L_{rm IR}^{rm scriptscriptstyle SF}/M_{rm dust}$ or $L_{160}^{rmscriptscriptstyle SF}/L_{70}^{rmscriptscriptstyle SF}$. In dusty star forming galaxies, the change in $L_{rm IR}^{rmscriptscriptstyle SF}/M_{rm dust}$ can fully account for the observed colder dust temperatures, suggesting that any change in the spatial extent of the interstellar medium is a second order effect.