Elastic $^{16}$O+$^{12}$C scattering is known to exhibit the nuclear rainbow pattern at incident energies $E_text{lab}gtrsim 200$ MeV, with the Airy structure of the far-side scattering cross section clearly seen at medium and large angles. Such a rainbow pattern is well described by the deep real optical potential (OP) given by the double-folding model (DFM). At lower energies, the extensive elastic $^{16}$O+$^{12}$C scattering data show consistently that the nuclear rainbow pattern at backward angles is deteriorated by an oscillating enhancement of elastic cross section that is difficult to describe in the conventional optical model (OM). Given a significant $alpha$ spectroscopic factor predicted for the dissociation $^{16}$O$toalpha+^{12}$C by the shell model and $alpha$-cluster models, the contribution of the elastic $alpha$ transfer (or the core-core exchange) to the elastic $^{16}$O+$^{12}$C scattering should not be negligible and is expected to account for the enhanced elastic cross section at backward angles. To reveal the impact of the elastic $alpha$ transfer, a systematic coupled reaction channels analysis of the elastic $^{16}$O+$^{12}$C scattering has been performed, with the coupling between the elastic scattering and elastic $alpha$ transfer channels treated explicitly, using the real OP given by the DFM. We found that the elastic $alpha$ transfer enhances the near-side scattering significantly at backward angles, giving rise to an oscillating distortion of the smooth Airy structure. The dynamic polarization of the OP by the coupling between the elastic scattering and elastic $alpha$ transfer channels can be effectively taken into account in the OM calculation by an angular-momentum (or parity) dependent potential added to the imaginary OP, as suggested by Frahn and Hussein 40 years ago.