We investigate spectral statistics in spatially extended, chaotic many-body quantum systems with a conserved charge. We compute the spectral form factor $K(t)$ analytically for a minimal Floquet circuit model that has a $U(1)$ symmetry encoded via auxiliary spin-$1/2$ degrees of freedom. Averaging over an ensemble of realizations, we relate $K(t)$ to a partition function for the spins, given by a Trotterization of the spin-$1/2$ Heisenberg ferromagnet. Using Bethe Ansatz techniques, we extract the Thouless time $t^{vphantom{*}}_{rm Th}$ demarcating the extent of random matrix behavior, and find scaling behavior governed by diffusion for $K(t)$ at $tlesssim t^{vphantom{*}}_{rm Th}$. We also report numerical results for $K(t)$ in a generic Floquet spin model, which are consistent with these analytic predictions.