In this work, we carry out first-principles calculations and lattice mode analysis to investigate the polarization switching mechanism in HfO$_2$. Because the stability of the polar orthorhombic $Pca2_1$ phase of HfO$_2$ arises from a trilinear coupling, polarization switching requires the flipping of not only the polar $Gamma_{15}^Z$ mode, but also at least one zone-boundary anti-polar mode. The coupling between the polar and anti-polar modes thus leads to substantial differences among different polarization switching paths. Specifically, our lattice-mode-coupling analysis shows that paths in which the $X_2^-$ mode is reversed involve a large activation energy, which because the $X_2^-$ mode is nonpolar cannot be directly overcome by applying an electric field. Our results show that the anti-polar $Pbca$ phase, whose structure is locally quite similar to that of the $Pca2_1$ phase, similarly cannot be transformed to this phase by an electric field as this would require local reversal of the $X_2^-$ mode pattern. Moreover, for the domain wall structure most widely considered, propagation also requires the reversal of the $X_2^-$ mode, leading to a much larger activation energy compared with that for the propagation of domain wall structures with a single sign for the $X_2^-$ mode. Finally, these first-principles results for domain wall propagation in HfO$_2$ have implications to many experimental observations, such as sluggish domain wall motion and robust ferroelectricity in thin films, and lattice mode analysis deepens our understanding of these distinctive properties of ferroelectric HfO$_2$.