Annual modulation of $gamma$ rays from ($alpha$, $gamma$) reactions in the Soudan Underground Lab has been observed using a 12-liter scintillation detector. This significant annual modulation, measured over 4 years, can mimic the signature for dark matter and can also generate potential background events for neutrinoless double-$beta$ decay experiments. The measured annual modulation of the event rate from ($alpha$, $gamma$) reactions is strongly correlated with the time-varying radon concentration observed independently in the Lab. The $alpha$ flux from radon decay is simulated starting from the measured radon concentration, and the $gamma$-ray flux is determined using the convolution of the $alpha$ flux and the cross sections for ($alpha$, $gamma$) reactions. The calculated $gamma$-ray flux is sufficient to generate the measured event rate that exhibits an annual modulation.