We present mid- and far- IR imaging of four famous hypergiant stars: the red supergiants $mu$ Cep and VY CMa, and the warm hypergiants IRC +10420 and $rho$ Cas. Our 11 to 37 $mu$m SOFIA/FORCAST imaging probes cool dust not detected in visual and near-IR imaging studies. Adaptive optics (AO) 8 - 12 $mu$m imaging of $mu$ Cep and IRC +10420 with MMT/MIRAC reveals extended envelopes that are the likely sources of these stars strong silicate emission features. We find $mu$ Ceps mass-loss rate to have declined by about a factor of 5 over a 13,000 history, ranging from 5 $times$ 10$^{-6}$ down to $sim$1 $times$ 10$^{-6}$ $M_{odot}$ yr$^{-1}$. The morphology of VY CMa indicates a cooler dust component coincident with the highly asymmetric reflection nebulae seen in the visual and near-IR. The lack of cold dust at greater distances around VY CMa indicates its mass-loss history is limited to the last $sim$1200 years, with an average rate of 6 $times$ 10$^{-4}$ $M_{odot}$ yr$^{-1}$. We find two distinct periods in the mass-loss history of IRC +10420 with a high rate of 2 $times$ 10$^{-3}$ $M_{odot}$ yr$^{-1}$ until approximately 2000 yr ago, followed by an order of magnitude decrease in the recent past. We interpret this change as evidence of its evolution beyond the RSG stage. Our new infrared photometry of $rho$ Cas is consistent with emission from the expanding dust shell ejected in its 1946 eruption, with no evidence of newer dust formation from its more recent events.