In this work, the anisotropic variant of the quantum Rabi model with different coupling strengths of the rotating and counter-rotating wave terms is studied by the Bogoliubov operator approach. The anisotropy preserves the parity symmetry of the original model. We derive the corresponding $G$-function, which yields both the regular and exceptional eigenvalues. The exceptional eigenvalues correspond to the crossing points of two energy levels with different parities and are doubly degenerate. We find analytically that the ground-state and the first excited state can cross several times, indicating multiple first-order phase transitions as function of the coupling strength. These crossing points are related to manifest parity symmetry of the Hamiltonian, in contrast to the level crossings in the asymmetric quantum Rabi model which are caused by a hidden symmetry.