While the quiet Sun magnetic field shows only little variation with the solar cycle, long-term variations cannot be completely ruled out from first principles. We investigate the potential effect of quiet Sun magnetism on spectral solar irradiance through a series of small-scale dynamo simulations with zero vertical flux imbalance ($langle B_zrangle=0$) and varying levels of small-scale magnetic field strength, and one weak network case with an additional flux imbalance corresponding to a flux density of $langle B_zrangle=100$ G. From these setups we compute the dependence of the outgoing radiative energy flux on the mean vertical magnetic field strength in the photosphere at continuum optical depth $tau=1$ ($langle vert B_zvertrangle_{tau=1}$). We find that a quiet Sun setup with a mean vertical field strength of $langle vert B_zvertrangle_{tau=1}=69$ G is about $0.6~%$ brighter than a non-magnetic reference case. We find a linear dependence of the outgoing radiative energy flux on the mean field strength $langle vert B_zvertrangle_{tau=1}$ with a relative slope of $1.4cdot 10^{-4}$ G$^{-1}$. With this sensitivity, only a moderate change of the quiet Sun field strength by $10%$ would lead to a total solar irradiance variation comparable to the observed solar cycle variation. While this does provide strong indirect constraints on possible quiet Sun variations during a regular solar cycle, it also emphasizes that potential variability over longer time scales could make a significant contribution to longer-term solar irradiance variations.