We study the structural evolution of isolated star-forming galaxies in the Illustris TNG100-1 hydrodynamical simulation, with a focus on investigating the growth of the central core density within 2 kpc ($Sigma_{*,2kpc}$) in relation to total stellar mass ($M_*$) at z < 0.5. First, we show that several observational trends in the $Sigma_{*,2kpc}$-$M_*$ plane are qualitatively reproduced in IllustrisTNG, including the distributions of AGN, star forming galaxies, quiescent galaxies, and radial profiles of stellar age, sSFR, and metallicity. We find that galaxies with dense cores evolve parallel to the $Sigma_{*,2kpc}$-$M_*$ relation, while galaxies with diffuse cores evolve along shallower trajectories. We investigate possible drivers of rapid growth in $Sigma_{*,2kpc}$ compared to $M_*$. Both the current sSFR gradient and the BH accretion rate are indicators of past core growth, but are not predictors of future core growth. Major mergers (although rare in our sample; $sim$10%) cause steeper core growth, except for high mass ($M_*$ >$sim$ $10^{10} M_{odot}$) mergers, which are mostly dry. Disc instabilities, as measured by the fraction of mass with Toomre Q < 2, are not predictive of rapid core growth. Instead, rapid core growth results in more stable discs. The cumulative black hole feedback history sets the maximum rate of core growth, preventing rapid growth in high-mass galaxies ($M_*$ >$sim$ $10^{9.5} M_{odot}$). For massive galaxies the total specific angular momentum of accreting gas is the most important predictor of future core growth. Our results suggest that the angular momentum of accreting gas controls the slope, width and zero-point evolution of the $Sigma_{*,2kpc}$-$M_*$ relation.