In this paper we explore the rich structure of macroscopic many-particle quantum states for Bose- Einstein condensate in an optical cavity with the tunable nonlinear atom-photon interaction [Nature (London) 464, 1301 (2010)]. Population inversion, bi
stable normal phases and the coexistence of normal{superradiant phases are revealed by adjusting of the experimentally realizable interaction strength and pump-laser frequency. For the negative (effective) cavity-frequency we observe remark- ably an inverted quantum phase transition (QPT) from the superradiant to normal phases with the increase of atom-field coupling, which is just opposite to the QPT in the normal Dicke model. The bistable macroscopic states are derived analytically in terms of the spin-coherent-state variational method by taking into account of both normal and inverted pseudospin states.
