Virgo is a dynamically young galaxy cluster with substructure in its spatial and kinematic distribution. Here, we simultaneously study the phase-space distribution and the main characteristics of Virgos galaxies, particularly its most abundant galaxy population - the early-type dwarfs - to understand their environmental transformation histories. Aside from known correlations with morphological types - like the larger average clustercentric distance of late-type galaxies - we find an intriguing behavior of early types with magnitudes -17 >= M_r >= -18. They show a large velocity spread and an asymmetric phase-space distribution, similar to the late-type galaxies and different from the early types just one magnitude brighter/fainter. Furthermore, we find a close phase-space aggregation of early-type dwarfs at large clustercentric distance and high relative velocity. Nearly all of them show signatures of disk components and their colors imply stellar ages that are younger than the population average. They are not located closely together but spread azimuthally around the cluster center. We show that this is expected from simulations of an infalling galaxy group that slowly gets dispersed after its first pericentric passage. We thus conclude that these galaxies are recent arrivals, and that the peculiar phase-space distribution of early-type dwarfs is evidence for the ongoing growth of this galaxy population. Studying galaxies based on their phase space correlations is a unique way to compare the properties of recent and older cluster members, and to understand which environment most influenced their present-day characteristics.