Various quantum phase transitions in the anisotropic Rabi-Stark model with both the nonlinear Stark coupling and the linear dipole coupling between a two-level system and a single-mode cavity are studied in this work. The first-order quantum phase transitions are detected by the level crossing of the ground-state and the first-excited state with the help of the pole structure of the transcendental functions derived by the Bogoliubov operators approach. As the nonlinear Stark coupling is the same as the cavity frequency, this model can be solved by mapping to an effective quantum oscillator. All energy levels close at the critical coupling in this case, indicating continuous quantum phase transitions. The critical gap exponent is independent of the anisotropy as long as the counter-rotating wave coupling is present, but essentially changed if the counter-rotating wave coupling disappears completely. It is suggested that the gapless Goldstone mode excitations could appear above a critical coupling in the present model in the rotating-wave approximation.