We present a comprehensive analysis of the ability of current stellar population models to reproduce the optical (ugriz) and near infra-red (JHK) colours of a small sample of well-studied nearby elliptical and S0 galaxies. We find broad agreement between the ages and metallicities derived using different population models, although different models show different systematic deviations from the measured broad-band fluxes. Although it is possible to constrain Simple Stellar Population models to a well defined area in age-metallicity space, there is a clear degeneracy between these parameters even with such a full range of precise colours. The precision to which age and metallicity can be determined independently, using only broad band photometry with realistic errors, is Delta{[Fe/H]} ~ 0.18 and Delta{log(Age)} ~ 0.25. To constrain the populations and therefore the star formation history further it will be necessary to combine broad-band optical-IR photometry with either spectral line indices, or else photometry at wavelengths outside of this range.