We present simultaneous multicolor infrared and optical photometry of the black hole X-ray transient XTE J1118+480 during its short 2005 January outburst, supported by simultaneous X-ray observations. The variability is dominated by short timescales, ~10s, although a weak superhump also appears to be present in the optical. The optical rapid variations, at least, are well correlated with those in X-rays. Infrared JHKs photometry, as in the previous outburst, exhibits especially large amplitude variability. The spectral energy distribution (SED) of the variable infrared component can be fitted with a power-law of slope alpha=-0.78 where Fnu is proportional to nu^alpha. There is no compelling evidence for evolution in the slope over five nights, during which time the source brightness decayed along almost the same track as seen in variations within the nights. We conclude that both short-term variability, and longer timescale fading, are dominated by a single component of constant spectral shape. We cannot fit the SED of the IR variability with a credible thermal component, either optically thick or thin. This IR SED is, however, approximately consistent with optically thin synchrotron emission from a jet. These observations therefore provide indirect evidence to support jet-dominated models for XTE J1118+480 and also provide a direct measurement of the slope of the optically thin emission which is impossible based on the average spectral energy distribution alone.