We present new metallicity measurements for 298 individual red giant branch stars in eight of the least luminous dwarf spheroidal galaxies (dSphs) in the Milky Way (MW) system. Our technique is based on medium resolution Keck/DEIMOS spectroscopy coupled with spectral synthesis. We present the first spectroscopic metallicities at [Fe/H] < -3.0 of stars in a dwarf galaxy, with individual stellar metallicities as low as [Fe/H] = -3.3. Because our [Fe/H] measurements are not tied to empirical metallicity calibrators and are sensitive to arbitrarily low metallicities, we are able to probe this extremely metal-poor regime accurately. The metallicity distribution of stars in these dSphs is similar to the MW halo at the metal-poor end. We also demonstrate that the luminosity-metallicity relation previously seen in more luminous dSph galaxies (M_V = -13.4 to -8.8) extends smoothly down to an absolute magnitude of M_V = -3.7. The discovery of extremely metal-poor stars in dSphs lends support to the LCDM galaxy assembly paradigm wherein dwarf galaxies dissolve to form the stellar halo of the MW.