We explore the collisional decay of disk mass and infrared emission in debris disks. With models, we show that the rate of the decay varies throughout the evolution of the disks, increasing its rate up to a certain point, which is followed by a leveling off to a slower value. The total disk mass falls off ~ t^-0.35 at its fastest point (where t is time) for our reference model, while the dust mass and its proxy -- the infrared excess emission -- fades significantly faster (~ t^-0.8). These later level off to a decay rate of M_tot(t) ~ t^-0.08 and M_dust(t) or L_ir(t) ~ t^-0.6. This is slower than the ~ t^-1 decay given for all three system parameters by traditional analytic models. We also compile an extensive catalog of Spitzer and Herschel 24, 70, and 100 micron observations. Assuming a log-normal distribution of initial disk masses, we generate model population decay curves for the fraction of debris disk harboring stars observed at 24 micron and also model the distribution of measured excesses at the far-IR wavelengths (70-100 micron) at certain age regimes. We show general agreement at 24 micron between the decay of our numerical collisional population synthesis model and observations up to a Gyr. We associate offsets above a Gyr to stochastic events in a few select systems. We cannot fit the decay in the far infrared convincingly with grain strength properties appropriate for silicates, but those of water ice give fits more consistent with the observations.