We present the results of realistic N-body modelling of massive star clusters in the Magellanic Clouds, aimed at investigating a dynamical origin for the radius-age trend observed in these systems. We find that stellar-mass black holes, formed in the supernova explosions of the most massive cluster stars, can constitute a dynamically important population. If a significant number of black holes are retained (here we assume complete retention), these objects rapidly form a dense core where interactions are common, resulting in the scattering of black holes into the cluster halo, and the ejection of black holes from the cluster. These two processes heat the stellar component, resulting in prolonged core expansion of a magnitude matching the observations. Significant core evolution is also observed in Magellanic Cloud clusters at early times. We find that this does not result from the action of black holes, but can be reproduced by the effects of mass-loss due to rapid stellar evolution in a primordially mass segregated cluster.