In the easy-plane regime of XXZ spin chains, spin transport is ballistic, with a Drude weight that has a discontinuous fractal dependence on the value of the anisotropy $Delta = cos pi lambda$ at nonzero temperatures. We show that this structure necessarily implies the divergence of the low-frequency conductivity for generic irrational values of $lambda$. Within the framework of generalized hydrodynamics, we show that in the high-temperature limit the low-frequency conductivity at a generic anisotropy scales as $sigma(omega) sim 1/sqrt{omega}$; anomalous response occurs because quasiparticles undergo Levy flights. For rational values of $lambda$, the divergence is cut off at low frequencies and the corrections to ballistic spin transport are diffusive. We also use our approach to recover that at the isotropic point $Delta=1$, spin transport is superdiffusive with $sigma(omega) sim omega^{-1/3}$. We support our results with extensive numerical studies using matrix-product operator methods.