Chandra observations of the Cartwheel galaxy reveal a population of ultraluminous X-ray sources (ULXs) with lifetimes < 10^7 yr associated with a spreading wave of star formation which began some 3 x 10^8 yr ago. A population of high-mass X-ray binaries provides a simple model: donor stars of initial masses M_2 > 15 Msun transfer mass on their thermal timescales to black holes of masses M_1 > 10 Msun. For alternative explanations of the Cartwheel ULX population in terms of intermediate-mass black holes (IMBH) accreting from massive stars, the inferred production rate > 10^-6 yr^-1 implies at least 300 IMBHs, and more probably 3 x 10^4, within the star-forming ring. These estimates are increased by factors eta^-1 if the efficiency eta with which IMBHs find companions of > 15 Msun within 10^7 yr is <1. Current models of IMBH production would require a very large mass ($ga 10^{10}msun$) of stars to have formed new clusters. Further, the accretion efficiency must be low (< 6 x 10^-3) for IMBH binaries, suggesting super-Eddington accretion, even though intermediate black hole masses are invoked with the purpose of avoiding it. These arguments suggest either that to make a ULX, an IMBH must accrete from some as yet unknown non-stellar mass reservoir with very specific properties, or that most if not all ULXs in star-forming galaxies are high-mass X-ray binaries.