We report on a novel experiment to generate non-classical atomic states via quantum non-demolition (QND) measurements on cold atomic samples prepared in a high finesse ring cavity. The heterodyne technique developed for the QND detection exhibits an
optical shot-noise limited behavior for local oscillator optical power of a few hundred muW, and a detection bandwidth of several GHz. This detection tool is used in single pass to follow non destructively the internal state evolution of an atomic sample when subjected to Rabi oscillations or a spin-echo interferometric sequence.
We investigate the quantum non-demolition (QND) measurement of an atomic population based on a heterodyne detection and show that the induced back-action allows to prepare both spin-squeezed and Dicke states. We use a wavevector formalism to describe
the stochastic process of the measurement and the associated atomic evolution. Analytical formulas of the atomic distribution momenta are derived in the weak coupling regime both for short and long time behavior, and they are in good agreement with those obtained by a Monte-Carlo simulation. The experimental implementation of the proposed heterodyne detection scheme is discussed. The role played in the squeezing process by the spontaneous emission is considered.
