The $mathrm{SU}(r)$ Vafa-Witten partition function, which virtually counts Higgs pairs on a projective surface $S$, was mathematically defined by Tanaka-Thomas. On the Langlands dual side, the first-named author recently introduced virtual counts of Higgs pairs on $mu_r$-gerbes. In this paper, we instead use Yoshiokas moduli spaces of twisted sheaves. Using Chern character twisted by rational $B$-field, we give a new mathematical definition of the $mathrm{SU}(r) / mathbb{Z}_r$ Vafa-Witten partition function when $r$ is prime. Our definition uses the period-index theorem of de Jong. $S$-duality, a concept from physics, predicts that the $mathrm{SU}(r)$ and $mathrm{SU}(r) / mathbb{Z}_r$ partitions functions are related by a modular transformation. We turn this into a mathematical conjecture, which we prove for all $K3$ surfaces and prime numbers $r$.