Planetary systems can survive the stellar evolution, as evidenced by the atmospheric metal pollution and dusty disks of single white dwarfs. Recent observations show that 1 to 4 percent of single white dwarfs are accompanied by dusty disks, while the occurrence rate of metal pollution is about 25 to 50 percent . The dusty disks and metal pollution have been associated with accretion of remnant planetary systems around white dwarfs, yet the relation between these two phenomena is still unclear. Here we suggest an evolutionary scenario to link the two observational phenomena. By analyzing a sample of metal polluted white dwarfs, we find that the mass accretion rate onto the white dwarf generally follows a broken power law decay, which matches well with the theoretical prediction, if assuming dust accretion is primarily driven by Poynting-Robertson drag and the dust source is primarily delivered via dynamically falling asteroids perturbed by a Jovian planet. The presence of disks is mainly at the early stage (about 0.1 to 0.7 billion years) of the whole process of metal pollution, which is detectable until about 8 billion years, naturally explaining the fraction (about 2-16 percent) of metal-polluted white dwarfs having dusty disks. The success of this scenario also implies that the configuration of an asteroid belt with an outer gas giant might be common around stars of several solar masses.