We quantify the calibration requirements for systematic uncertainties for next-generation ground-based observatories targeting the large-angle $B$-mode polarization of the Cosmic Microwave Background, with a focus on the Simons Observatory (SO). We explore uncertainties on gain calibration, bandpass center frequencies, and polarization angles, including the frequency variation of the latter across the bandpass. We find that gain calibration and bandpass center frequencies must be known to percent levels or less to avoid biases on the tensor-to-scalar ratio $r$ on the order of $Delta rsim10^{-3}$, in line with previous findings. Polarization angles must be calibrated to the level of a few tenths of a degree, while their frequency variation between the edges of the band must be known to ${cal O}(10)$ degrees. Given the tightness of these calibration requirements, we explore the level to which residual uncertainties on these systematics would affect the final constraints on $r$ if included in the data model and marginalized over. We find that the additional parameter freedom does not degrade the final constraints on $r$ significantly, broadening the error bar by ${cal O}(10%)$ at most. We validate these results by reanalyzing the latest publicly available data from the BICEP2/Keck collaboration within an extended parameter space covering both cosmological, foreground and systematic parameters. Finally, our results are discussed in light of the instrument design and calibration studies carried out within SO.