We prove that the direct image complex for the $D$-twisted $SL_n$ Hitchin fibration is determined by its restriction to the elliptic locus, where the spectral curves are integral. The analogous result for $GL_n$ is due to P.-H. Chaudouard and G. Laumon. Along the way, we prove that the Tate module of the relative Prym group scheme is polarizable, and we also prove $delta$-regularity results for some auxiliary weak abelian fibrations.