We benchmark angular-momentum projected Hartree-Fock calculations as an approximation to full configuration-interaction results in a shell model basis. For such a simple approximation we find reasonably good agreement between excitation spectra, including for many odd-$A$ and odd-odd nuclides. We frequently find shape coexistence, in the form of multiple Hartree-Fock minima, which demonstrably improves the spectrum in the $sd$- and $pf$-shells. The complex spectra of germanium isotopes present a challenge: for even $A$ the spectra are only moderately good and those of odd $A$ bear little resemblance to the configuration-interaction results. Despite this failure we are able to broadly reproduce the odd-even staggering of ground state binding energies, save for germanium isotopes with $N > 40$. To illustrate potential applications, we compute the spectrum of the recently measured dripline nuclide $^{40}$Mg. All in all, projected Hartree-Fock often provides a better description of low-lying nuclear spectra than one might expect. Key to this is the use of gradient descent and unrestricted shapes.