We study the geometric curvature and phase of the Rabi model. Under the rotating-wave approximation (RWA), we apply the gauge independent Berry curvature over a surface integral to calculate the Berry phase of the eigenstates for both single and two-qubit systems, which is found to be identical with the system of spin-1/2 particle in a magnetic field. We extend the idea to define a vacuum-induced geometric curvature when the system starts from an initial state with pure vacuum bosonic field. The induced geometric phase is related to the average photon number in a period which is possible to measure in the qubit-cavity system. We also calculate the geometric phase beyond the RWA and find an anomalous sudden change, which implies the breakdown of the adiabatic theorem and the Berry phases in an adiabatic cyclic evolution are ill-defined near the anti-crossing point in the spectrum.