We experimentally investigate the dynamical instability of a Bose Einstein condensate in an optical ring resonator for various cavity detuning and pump powers. The resulting phase diagram is asymmetric with respect to the cavity detuning and can be d
escribed by the coupling of two atomic modes with one optical mode. We compare the experimental data to a numerical simulation and to an analytic expression of the phase boundary. For positive and negative pump cavity detuning different coupling mechanisms are identified explaining the asymmetry of the phase diagram. We present a physical interpretation and discuss the connection to the Dicke quantum phase transition.
