We present an analysis of the spatial distribution of various stellar populations within the Large and Small Magellanic Clouds. We use optically selected stellar samples with mean ages between ~9 and ~1000 Myr, and existing stellar cluster catalogues to investigate how stellar structures form and evolve within the LMC/SMC. We use two statistical techniques to study the evolution of structure within these galaxies, the $Q$-parameter and the two-point correlation function (TPCF). In both galaxies we find the stars are born with a high degree of substructure (i.e. are highly fractal) and that the stellar distribution approaches that of the background population on timescales similar to the crossing times of the galaxy (~80/150 Myr for the SMC/LMC respectively). By comparing our observations to simple models of structural evolution we find that popping star clusters do not significantly influence structural evolution in these galaxies. Instead we argue that general galactic dynamics are the main drivers, and that substructure will be erased in approximately the crossing time, regardless of spatial scale, from small clusters to whole galaxies. This can explain why many young Galactic clusters have high degrees of substructure, while others are smooth and centrally concentrated. We conclude with a general discussion on cluster infant mortality, in an attempt to clarify the time/spatial scales involved.