Measurements from the Solar Irradiance Monitor (SIM) onboard the SORCE mission indicate that solar spectral irradiance at Visible and IR wavelengths varies in counter phase with the solar activity cycle. The sign of these variations is not reproduced by most of the irradiance reconstruction techniques based on variations of surface magnetism employed so far, and it is not clear yet whether SIM calibration procedures need to be improved, or if instead new physical mechanisms must be invoked to explain such variations. We employ three-dimensional magneto hydrodynamic simulations of the solar photosphere to investigate the dependence of solar radiance in SIM Visible and IR spectral ranges on variations of the filling factor of surface magnetic fields. We find that the contribution of magnetic features to solar radiance is strongly dependent on the location on the disk of the features, being negative close to disk center and positive toward the limb. If features are homogeneously distributed over a region around the equator (activity belt) then their contribution to irradiance is positive with respect to the contribution of HD snapshots, but decreases with the increase of their magnetic flux for average magnetic flux larger than 50 G in at least two of the Visible and IR spectral bands monitored by SIM. Under the assumption that the 50 G snapshots are representative of quiet Sun regions we find thus that the Spectral Irradiance can be in counter-phase with the solar magnetic activity cycle.