Dwarf spheroidal galaxies are the smallest known stellar systems where under Newtonian interpretations, a significant amount of dark matter is required to explain observed kinematics. In fact, they are in this sense the most heavily dark matter dominated objects known. That, plus the increasingly small sizes of the newly discovered ultra faint dwarfs, puts these systems in the regime where dynamical friction on individual stars starts to become relevant. We calculate the dynamical friction timescales for pressure supported isotropic spherical dark matter dominated stellar systems, yielding $tau_{DF} =0.93 (r_{h}/10 pc)^{2} (sigma/ kms^{-1}) Gyr$, { where $r_{h}$ is the half-light radius}. For a stellar velocity dispersion value of $3 km/s$, as typical for the smallest of the recently detected ultra faint dwarf spheroidals, dynamical friction timescales becomes smaller than the $10 Gyr$ typical of the stellar ages for these systems, for $r_{h}<19 pc$. Thus, this becomes a theoretical lower limit below which dark matter dominated stellar systems become unstable to dynamical friction. We present a comparison with structural parameters of the smallest ultra faint dwarf spheroidals known, showing that these are already close to the stability limit derived, any future detection of yet smaller such systems would be inconsistent with a particle dark matter hypothesis.