While many observed ultra-luminous X-ray sources (ULXs, Lx > 10^39 erg s^-1) could be extragalactic X-ray binaries (XRBs) emitting close to the Eddington limit, the highest-luminosity ULXs (Lx > 3x10^39 erg s^-1) exceed the isotropic Eddington luminosity for even high-stellar-mass accreting black hole XRBs. It has been suggested that these highest-luminosity ULXs may contain accreting intermediate-mass black hole (IMBH) binaries. We consider this hypothesis for dense, young (about 100 Myr) stellar clusters where we assume that a 50-500 solar mass central IMBH has formed through runaway growth of a massive star. Using numerical simulations of the dynamics and evolution of the central black holes captured companions, we obtain estimates of the incidence of mass transfer phases and possible ULX activity throughout the IMBHs evolutionary history. We find that, although it is common for the central black hole to acquire binary companions, there is a very low probability that these interacting binaries will become observable ULX sources.