The mid- to far-infrared emission of the outbursting FU Orionis objects has been attributed either to a flared outer disk or to an infalling envelope. We revisit this issue using detailed radiative transfer calculations to model the recent, high signal-to-noise data from the IRS instrument on the {Spitzer Space Telescope}. In the case of FU Ori, we find that a physically-plausible flared disk irradiated by the central accretion disk matches the observations. Building on our previous work, our accretion disk model with outer disk irradiation by the inner disk reproduces the spectral energy distribution between ~4000 angstroms to ~40 microns. Our model is consistent with near-infrared interferometry but there are some inconsistencies with mid-infared interferometric results. Including the outer disk allows us to refine our estimate of the outer radius of the outbursting, high mass accretion rate disk in FU Ori as ~ 0.5 AU, which is a crucial parameter in assessing theories of the FU Orionis phenomenon. We are able to place an upper limit on the mass infall rate of any remnant envelope infall rate to ~ 7e-7 Msun/yr assuming a centrifugal radius of 200 AU. The FUor BBW 76 is also well modelled by a 0.6 AU inner disk and a flared outer disk. However, V1515 Cyg requires an envelope with an outflow cavity to adequately reproduce the IRS spectrum. In contrast with the suggestion by Green et al., we do not require a flattened envelope to match the observations; the inferred cavity shape is qualitatively consistent with typical protostellar envelopes. This variety of dusty structures suggests that the FU Orionis phase can be present at either early or late stages of protostellar evolution.