The morphological asymmetry of leading and following sunspots is a well-known characteristic of the solar surface. In the context of large-scale evolution of the surface magnetic field, the asymmetry has been assumed to have only a negligible effect. Using the surface flux transport model, we show that the morphological asymmetry of leading and following sunspots has a significant impact on the evolution of the large-scale magnetic field on the solar surface. By evaluating the effect of the morphological asymmetry of each bipolar magnetic region (BMR), we observe that the introduction of the asymmetry in the BMR model significantly reduces its contribution to the polar magnetic field, especially for large and high-latitude BMRs. Strongly asymmetric BMRs can even reverse the regular polar field formation. The surface flux transport simulations based on the observed sunspot record shows that the introduction of the morphological asymmetry reduces the root-mean-square difference from the observed axial dipole strength by 30--40 percent. These results indicate that the morphological asymmetry of leading and following sunspots has a significant effect on the solar cycle prediction.