We present first-principles many-body perturbation theory calculations of the quasiparticle electronic structure and of the optical response of HfO$_2$ polymorphs. We use the $GW$ approximation including core electrons by the projector augmented wave (PAW) method and performing a quasiparticle self-consistency also on wavefunctions (QS$GW$). In addition, we solve the Bethe-Salpeter equation on top of $GW$ to calculate optical properties including excitonic effects. For monoclinic HfO$_2$ we find a fundamental band gap of $E_g = 6.33$ eV (with the direct band gap at $E_g^d = 6.41$ eV), and an exciton binding energy of 0.57 eV, which situates the optical gap at $E^o_g = 5.85$ eV. The latter is in the range of spectroscopic ellipsometry (SE) experimental estimates (5.5-6 eV), whereas our electronic band gap is well beyond experimental photoemission (PE) estimates ($< 6$ eV) and previous $GW$ works. Our calculated density of states and optical absorption spectra compare well to raw PE and SE spectra. This suggests that our predictions of both optical and electronic gaps are close to, or at least lower bounds of, the real values.