The value of residual phase noise, after phase compensation, is one of the key limitations of performance improvement for continuous-variable quantum key distribution using a local local oscillator (LLO CV-QKD) system, since it is the major excess noise. However, due to the non-ideality of the realistic devices implemented in practice, for example, imperfect lasers, detectors and unbalanced interferometers, the value of residual phase noise in current system is still relatively large. Here, we develop a phase noise model to improve the phase noise tolerance of the LLO CV-QKD schemes. In our model, part of the phase-reference measurement noise associated with detection efficiency and electronic noise of Bobs detector as well as a real-time monitored phasereference intensity at Bobs side is considered trusted because it can be locally calibrated by Bob. We show that using our phase noise model can significantly improve the secure key rate and transmission distance of the LLO CV-QKD system. We further conduct an experiment to substantiate the superiority of the phase noise model. Based on experimental data of a LLO CV-QKD system in the 25 km optical fiber channel, we demonstrate that the secure key rate under our phase noise model is approximately 40% higher than that under the conventional phase noise model.