The controversy over whether ultraluminous X-ray sources (ULXs) contain a new intermediate-mass class of black holes (IMBHs) remains unresolved. We present new analyses of the deepest XMM-Newton observations of ULXs that address their underlying nature. We examine both empirical and physical modelling of the X-ray spectra of a sample of thirteen of the highest quality ULX datasets, and find that there are anomalies in modelling ULXs as accreting IMBHs with properties simply scaled-up from Galactic black holes. Most notably, spectral curvature above 2 keV in several sources implies the presence of an optically-thick, cool corona. We also present a new analysis of a 100 ks observation of Holmberg II X-1, in which a rigorous analysis of the temporal data limits the mass of its black hole to no more than 100 solar masses. We argue that a combination of these results points towards many (though not necessarily all) ULXs containing black holes that are at most a few 10s of solar mass in size.