We adopt a three-level bosonic model to investigate the quantum phase transition in an ultracold atom-molecule conversion system which includes one atomic mode and two molecular modes. Through thoroughly exploring the properties of energy level structure, fidelity, and adiabatical geometric phase, we confirm that the system exists a second-order phase transition from an atommolecule mixture phase to a pure molecule phase. We give the explicit expression of the critical point and obtain two scaling laws to characterize this transition. In particular we find that both the critical exponents and the behaviors of ground-state geometric phase change obviously in contrast to a similar two-level model. Our analytical calculations show that the ground-state geometric phase jumps from zero to ?pi/3 at the critical point. This discontinuous behavior has been checked by numerical simulations and it can be used to identify the phase transition in the system.