Energy savings from efficiency methods in individual residential buildings are measured in 10s of dollars, while the energy savings from such measures nationally would amount to 10s of billions of dollars, leading to the tragedy of the commons effect. The way out of this situation is via deployment of automated, integrated residential energy systems, that provide the user with a seamless, cost effective service leading to improvement of comfort and residential experience. Models are of critical importance in this context, as intelligent operating systems depend on them strongly. However, most of the currently used models of thermal behavior of buildings have high complexity leading to problems and implementation. The complexity also obscures the utilization of well know physical properties of buildings such as the thermal mass. In view of this, we investigate data-driven, simple-to-implement residential environmental models that can serve as the basis for energy saving algorithms in both retrofits and new designs of residential buildings. Despite the nonlinearity of the underlying dynamics, using Koopman operator theory framework in this study we show that a linear second order model embedding, that captures the physics that occur inside a single or multi-zone space does well when compared with data simulated using EnergyPlus. This class of models has low complexity. We show that their parameters have physical significance for the large-scale dynamics of a building and are correlated to concepts such as the thermal mass. We investigate consequences of changing the thermal mass on the energy behavior of a building system and provide best practice design suggestions.