Let $X$ be a compact orientable non-Haken 3-manifold modeled on the Thurston geometry $text{Nil}$. We show that the diffeomorphism group $text{Diff}(X)$ deformation retracts to the isometry group $text{Isom}(X)$. Combining this with earlier work by many authors, this completes the determination the homotopy type of $text{Diff}(X)$ for any compact, orientable, prime 3-manifold $X$.