The photon emission by an ultrarelativistic charged particle in extremely strong magnetic field is analyzed, with vacuum polarization and photon recoil taken into account. The vacuum polarization is treated phenomenologically via refractive index. The photon emission occurs in the synergic (cooperative) synchrotron-Cherenkov process [J. Schwinger, W. Tsai and T. Erber, Annals of Physics, 96 303 (1976)] which is similar to the synchrotron emission rather than to the Cherenkov one. For electrons, the effect of the vacuum polarization on the emission spectrum is not evident even beyond the probable onset of non-perturbative quantum electrodynamics (QED). However, the effect of the vacuum polarization on the emission spectrum can be observable for muons already at $gamma B / B_S approx 30$, with $gamma$ the muon Lorentz factor, $B$ the magnetic field strength and $B_S$ the critical QED field. Nevertheless, vacuum polarization leads to only 10% enhancement of the maximum of the radiation spectrum.