Stellar associations can be discerned as overdensities of sources not only in the physical space but also in the velocity space. The common motion of their members, gradually eroded by the galactic tidal field, is partially reminiscent of the initial kinematic structure. Using recent data from Gaia EDR3, combined with radial velocities from GALAH and APOGEE, we traced back the present positions of stars belonging to Upper Scorpius, a subgroup of Scorpius-Centaurus, the nearest OB association. About one half of the subgroup (the clustered population) appears composed of many smaller entities, which were in a more compact configuration in the past. The presence of a kinematic duality is reflected into an age spread between this younger clustered population and an older diffuse population, in turn confirmed by a different fraction $f_D$ of disc-bearing stars ($f_D = 0.24pm0.02$ vs $f_D = 0.10pm 0.01$). Star formation in Upper Scorpius appears to have lasted more than 10 Myr and proceeded in small groups that, after a few Myr, dissolve in the field of the older population but retain for some time memory of their initial structure. The difference of ages inferred through isochrones and kinematics, in this regard, could provide a powerful tool to quantify the timescale of gas removal.