One of the important open questions in solar irradiance studies is whether long-term variability (i.e. on timescales of years and beyond) can be reconstructed by means of models that describe short-term variability (i.e. days) using solar proxies as inputs. Preminger and Walton (2005, GRL, 32, 14109) showed that the relationship between spectral solar irradiance and proxies of magnetic-flux emergence, such as the daily sunspot area, can be described in the framework of linear system theory by means of the impulse response. We significantly refine that empirical model by removing spurious solar-rotational effects and by including an additional term that captures long-term variations. Our results show that long-term variability cannot be reconstructed from the short-term response of the spectral irradiance, which cautions the extension of solar proxy models to these timescales. In addition, we find that the solar response is nonlinear in such a way that cannot be corrected simply by applying a rescaling to sunspot area.