We present near and mid-infrared interferometric observations made with the Keck Interferometer Nuller and near-contemporaneous spectro-photometry from the IRTF of 11 well known young stellar objects, several observed for the first time in these spectral and spatial resolution regimes. With AU-level spatial resolution, we first establish characteristic sizes of the infrared emission using a simple geometrical model consisting of a hot inner rim and mid-infrared disk emission. We find a high degree of correlation between the stellar luminosity and the mid-infrared disk sizes after using near-infrared data to remove the contribution from the inner rim. We then use a semi-analytical physical model to also find that the very widely used star + inner dust rim + flared disk class of models strongly fails to reproduce the SED and spatially-resolved mid-infrared data simultaneously; specifically a more compact source of mid-infrared emission is required than results from the standard flared disk model. We explore the viability of a modification to the model whereby a second dust rim containing smaller dust grains is added, and find that the two-rim model leads to significantly improved fits in most cases. This complexity is largely missed when carrying out SED modelling alone, although detailed silicate feature fitting by McClure et al. 2013 recently came to a similar conclusion. As has been suggested recently by Menu et al. 2015, the difficulty in predicting mid-infrared sizes from the SED alone might hint at transition disk-like gaps in the inner AU; however, the relatively high correlation found in our mid-infrared disk size vs. stellar luminosity relation favors layered disk morphologies and points to missing disk model ingredients instead.