The phase of a quantum state may not return to its original value after the systems parameters cycle around a closed path; instead, the wavefunction may acquire a measurable phase difference called the Berry phase. Berry phases typically have been accessed through interference experiments. Here, we demonstrate an unusual Berry-phase-induced spectroscopic feature: a sudden and large increase in the energy of angular-momentum states in circular graphene p-n junction resonators when a small critical magnetic field is reached. This behavior results from turning on a $pi$-Berry phase associated with the topological properties of Dirac fermions in graphene. The Berry phase can be switched on and off with small magnetic field changes on the order of 10 mT, potentially enabling a variety of optoelectronic graphene device applications.