(Abbreviated) We show that the XRT spectral response calibration was complicated by various energy offsets in photon counting (PC) and windowed timing (WT) modes related to the way the CCD is operated in orbit (variation in temperature during observations, contamination by optical light from the sunlit Earth and increase in charge transfer inefficiency). We describe how these effects can be corrected for in the ground processing software. We show that the low-energy response, the redistribution in spectra of absorbed sources, and the modelling of the line profile have been significantly improved since launch by introducing empirical corrections in our code when it was not possible to use a physical description. We note that the increase in CTI became noticeable in June 2006 (i.e. 14 months after launch), but the evidence of a more serious degradation in spectroscopic performance (line broadening and change in the low-energy response) due to large charge traps (i.e. faults in the Si crystal) became more significant after March 2007. We describe efforts to handle such changes in the spectral response. Finally, we show that the commanded increase in the substrate voltage from 0 to 6V on 2007 August 30 reduced the dark current, enabling the collection of useful science data at higher CCD temperature (up to -50C). We also briefly describe the plan to recalibrate the XRT response files at this new voltage.