Efficient frequency conversion of photons has important applications in optical quantum technology because the frequency range suitable for photon manipulation and communication usually varies widely. Recently, an efficient frequency conversion system using a double-$Lambda$ four-wave mixing (FWM) process based on electromagnetically induced transparency (EIT) has attracted considerable attention because of its potential to achieve a nearly 100% conversion efficiency (CE). To obtain such a high CE, the spontaneous emission loss in this resonant-type FWM system must be suppressed considerably. A simple solution is to arrange the applied laser fields in a backward configuration. However, the phase mismatch due to this configuration can cause a significant decrease in CE. Here, we demonstrate that the phase mismatch can be effectively compensated by introducing the phase shift obtained by two-photon detuning. Under optimal conditions, we observe a wavelength conversion from 780 to 795 nm with a maximum CE of 91.2(6)% by using this backward FWM system at an optical depth of 130 in cold rubidium atoms. The current work represents an important step toward achieving low-loss, high-fidelity EIT-based quantum frequency conversion.