We predict that a strong nonreciprocity in the resonance spectra of Dirac quantum dots can be induced by the Berry phase. The nonreciprocity arises in relatively weak magnetic fields and is manifest in anomalously large field-induced splittings of quantum dot resonances which are degenerate at $B=0$ due to time-reversal symmetry. This exotic behavior, which is governed by field-induced jumps in the Berry phase of confined electronic states, is unique to quantum dots in Dirac materials and is absent in conventional quantum dots. The effect is strong for gapless Dirac particles and can overwhelm the $B$-induced orbital and Zeeman splittings. A finite Dirac mass suppresses the effect. The nonreciprocity, predicted for generic two-dimensional Dirac materials, is accessible through Faraday and Kerr optical rotation measurements and scanning tunneling spectroscopy.