CONTEXT: The dynamical mass-to-light (M/L) ratios of massive ultra-compact dwarf galaxies (UCDs) are about 50% higher than predicted by stellar population models. AIMS: Here we investigate the possibility that these elevated M/L ratios are caused by a central black hole (BH), heating up the internal motion of stars. We focus on a sample of ~50 extragalactic UCDs for which velocity dispersions and structural parameters have been measured. METHODS: Using up-to-date distance moduli and a consistent treatment of aperture and seeing effects, we calculate the ratio Psi=(M/L)_{dyn}/(M/L)_{pop} between the dynamical and the stellar population M/L of UCDs. For all UCDs with Psi>1 we estimate the mass of a hypothetical central BH needed to reproduce the observed integrated velocity dispersion. RESULTS: Massive UCDs (M>10^7 M_*) have an average Psi = 1.7 +-0.2, implying notable amounts of dark mass in them. We find that, on average, central BH masses of 10-15% of the UCD mass can explain these elevated dynamical M/L ratios. The implied BH masses in UCDs range from several 10^5 M_* to several 10^7 M_*. In the M_BH-Luminosity plane, UCDs are offset by about two orders of magnitude in luminosity from the relation derived for galaxies. Our findings can be interpreted such that massive UCDs originate from progenitor galaxies with masses around 10^9 M_*, and that those progenitors have SMBH occupation fractions of 60-100%. The suggested UCD progenitor masses agree with predictions from the tidal stripping scenario. Lower-mass UCDs (M<10^7 M_*) exhibit a bimodal distribution in Psi, suggestive of a coexistence of massive globular clusters and tidally stripped galaxies in this mass regime. CONCLUSIONS: Central BHs as relict tracers of tidally stripped progenitor galaxies are a plausible explanation for the elevated dynamical M/L ratios of UCDs.