Strong spectral softening has been revealed in the late X-ray afterglows of some gamma-ray bursts (GRBs). The scenario of X-ray scattering around circum-burst dusty medium has been supported by previous works due to its overall successful prediction of both the temporal and spectral evolution of some X-ray afterglows. To further investigate the observed feature of spectral softening, we now systematically search the X-ray afterglows detected by X-Ray Telescope (XRT) of Swift and collect twelve GRBs with significant late-time spectral softening. We find that dust scattering could be the dominant radiative mechanism for these X-ray afterglows regarding their temporal and spectral features. For some well observed bursts with high-quality data, their time-resolved spectra could be well produced within the scattering scenario by taking into account the X-ray absorption from circum-burst medium. We also find that during spectral softening the power-law index in the high energy end of the spectra does not vary much. The spectral softening is mainly manifested by the spectral peak energy continually moving to the soft end.