In this paper, the nonlinear mode conversion of extraordinary waves in nonuniform magnetized plasmas is studied using the variational symplectic particle-in-cell simulation. The accuracy of the nonlinear simulation is guaranteed by the long-term accuracy and conservativeness of the symplectic algorithm. The spectra of the electromagnetic wave, the evolution of the wave reflectivity, the energy deposition profile, and the parameter-dependent properties of radio-frequency waves during the nonlinear mode conversion are investigated. It is illustrated that nonlinear effects significantly modify the physics of the radio-frequency injection in magnetized plasmas. The evolutions of the radio-frequency wave reflectivity and the energy deposition are observed, as well as the self-interaction of the Bernstein waves and mode excitations. Even for waves with small magnitude, nonlinear effects can also become important after continuous wave injections, which are common in the realistic radio-frequency wave heating and current drive experiments.