We analyse the dynamical properties of substructures in a high-resolution dark matter simulation of the formation of a Milky Way-like halo in a $Lambda$CDM cosmology. Our goal is to shed light on the dynamical peculiarities of the Milky Way satellites. Our simulations show that about 1/3 of the subhalos have been accreted in groups. We quantify this clustering by measuring the alignment of the angular momentum of subhalos in a group. We find that this signal is visible even for objects accreted up to $z sim 1$, i.e. 8 Gyr ago, and long after the spatial coherence of the groups has been lost due the host tidal field. This group infall may well explain the ghostly streams proposed by Lynden-Bell & Lynden-Bell to orbit the Milky Way. Our analyses also show that if most satellites originate in a few groups, the disk-like distribution of the Milky Way satellites would be almost inevitable. This non-random assignment of satellites to subhalos implies an environmental dependence on whether these low-mass objects are able to form stars, possibly related to the nature of reionization in the early Universe. With this picture, both the ``ghostly streams and the ``disk-like configuration are manifestations of the same phenomenon: the hierarchical growth of structure down to the smallest scales.