Beam-target double-spin asymmetries and target single-spin asymmetries were measured for the exclusive $pi^0$ electroproduction reaction $gamma^* p to p pi^0$, expanding an analysis of the $gamma^* p to n pi^+$ reaction from the same experiment. The results were obtained from scattering of 6 GeV longitudinally polarized electrons off longitudinally polarized protons using the CEBAF Large Acceptance Spectrometer at Jefferson Lab. The kinematic range covered is $1.1<W<3$ GeV and $1<Q^2<6$ GeV$^2$. Results were obtained for about 5700 bins in $W$, $Q^2$, cthcm, and $phi^*$. The beam-target asymmetries were found to generally be greater than zero, with relatively modest phicmsp dependence. The target asymmetries exhibit very strong phicmsp dependence, with a change in sign occurring between results at low $W$ and high $W$, in contrast to $pi^+$ electroproduction. Reasonable agreement is found with phenomenological fits to previous data for $W<1.6$ GeV, but significant differences are seen at higher $W$. When combined with cross section measurements, as well as $pi^+$ observables, the present results will provide powerful constraints on nucleon resonance amplitudes at moderate and large values of $Q^2$, for resonances with masses as high as 2.4 GeV.