We investigate the origin of the evolution of the population-averaged central stellar mass density ($Sigma_1$) of quiescent galaxies (QGs) by probing the relation between stellar age and $Sigma_1$ at $zsim0$. We use the Zurich ENvironmental Study (ZENS), which is a survey of galaxy groups with a large fraction of satellite galaxies. QGs shape a narrow locus in the $Sigma_1-M_{star}$ plane, which we refer to as $Sigma_1$ ridgeline. Colors of ($B-I$) and ($I-J$) are used to divide QGs into three age categories: young ($<2~mathrm{Gyr}$), intermediate ($2-4~mathrm{Gyr}$), and old ($>4~mathrm{Gyr}$). At fixed stellar mass, old QGs on the $Sigma_1$ ridgeline have higher $Sigma_1$ than young QGs. This shows that galaxies landing on the $Sigma_1$ ridgeline at later epochs arrive with lower $Sigma_1$, which drives the zeropoint of the ridgeline down with time. We compare the present-day zeropoint of the oldest population at $z=0$ with the zeropoint of the quiescent population 4 Gyr back in time, at $z=0.37$. These zeropoints are identical, showing that the intrinsic evolution of individual galaxies after they arrive on the $Sigma_1$ ridgeline must be negligible, or must evolve parallel to the ridgeline during this interval. The observed evolution of the global zeropoint of 0.07 dex over the last 4 Gyr is thus largely due to the continuous addition of newly quenched galaxies with lower $Sigma_1$ at later times (progenitor bias). While these results refer to the satellite-rich ZENS sample as a whole, our work suggests a similar age-$Sigma_1$ trend for central galaxies.