We determine the ensemble properties of z~5 Lyman break galaxies (LBGs) selected as V-band dropouts to i(AB)<26.3 in the Chandra Deep Field South using their rest-frame UV-to-visible SEDs. By matching the selection and performing the same analysis that has been used for z~3 samples, we show clear differences in the properties of two samples of LBGs which are separated by ~1Gyr in lookback time. We find that z~5 LBGs are typically much younger (<100Myr) and have lower stellar masses (10^9Msol) than their z~3 counterparts. The difference in mass is significant even when considering the presence of an older, underlying population in both samples. Such young and moderately massive systems dominate the luminous z~5 LBG population (>70%), whereas they comprise <30% of LBG samples at z~3. This result is robust under all reasonable modelling assumptions. These intense starbursts appear to be experiencing their first (few) generations of large-scale star formation and are accumulating their first significant stellar mass. Their dominance in luminous LBG samples suggests that z~5 witnesses a period of wide-spread, recent galaxy formation. As such, z~5 LBGs are the likely progenitors of the spheroidal components of present-day massive galaxies. This is supported by their high stellar mass surface densities, their core phase-space densities, as well as the ages of stars in the bulge of our Galaxy and other massive systems. Their high star formation rates per unit area suggest that these systems host outflows or winds that enrich the intra- and inter-galactic media with metals. Their estimated young ages are consistent with inefficient metal-mixing on galaxy-wide scales. Therefore these galaxies may contain a significant fraction of metal-free stars as has been proposed for z~3 LBGs (Jimenez & Haiman 2006). [Abridged]