We explore the potential of direct spectroscopy of ultra-narrow optical transitions of atoms localized in an optical cavity. In contrast to stabilization against a reference cavity, which is the approach currently used for the most highly stabilized lasers, stabilization against an atomic transition does not suffer from Brownian thermal noise. Spectroscopy of ultra-narrow optical transitions in a cavity operates in a very highly saturated regime in which non-linear effects such as bistability play an important role. From the universal behavior of the Jaynes-Cummings model with dissipation, we derive the fundamental limits for laser stabilization using direct spectroscopy of ultra-narrow atomic lines. We find that with current lattice clock experiments, laser linewidths of about 1 mHz can be achieved in principle, and the ultimate limitations of this technique are at the 1 $mu$ Hz level.