We provide a comprehensive description and offer an explanation for the sizes of the faintest known galaxies in the universe, the dwarf spheroidal (dSph) satellites of the Milky Way and Andromeda. After compiling a consistent data set of half-light radii (r_{1/2}) and luminosities, we describe the size-luminosity relation of dSphs by a log-normal distribution in r_{1/2} with a mean size that varies as a function of luminosity. Accounting for modest number statistics, measurement uncertainties and surface brightness limitations, we find that the size-luminosity relations of the Milky Way and Andromeda dSph populations are statistically indistinguishable, and also very similar: their mean sizes at a given stellar luminosity differ by no more than 30%. In addition, we find that the mean size, slope and scatter of this log-normal size description of Local Group dSphs matches onto the relation of more massive low-concentration galaxies. This suggests that the stellar sizes of dSphs are ultimately related to their overall initial baryonic angular momentum. To test this hypothesis we perform a series of high resolution N-body simulations that we couple with a semi-analytic model of galaxy formation. These predict the same mean size and slope as observed in dSph satellites. At the same time, these models predict that the size-luminosity distributions for satellite galaxies around similar host-halos must be similar providing a natural explanation as to why the size distributions of Milky Way and Andromeda satellites are similar. Although strong rotation is currently not observed in dSphs, this may well be consistent with our angular-momentum-based explanation for their sizes if the disks of these galaxies have become sufficiently stirred through tidal interaction.