We measure stellar masses and structural parameters for 5,500 quiescent and 20,000 star-forming galaxies at 0.3<zleq1.5 in the Newfirm Medium Band Survey COSMOS and UKIDSS UDS fields. We combine these measurements to infer velocity dispersions and determine how the number density of galaxies at fixed inferred dispersion, or the Velocity Dispersion Function (VDF), evolves with time for each population. We show that the number of galaxies with high velocity dispersions appears to be surprisingly stable with time, regardless of their star formation history. Furthermore, the overall VDF for star-forming galaxies is constant with redshift, extending down to the lowest velocity dispersions probed by this study. The only galaxy population showing strong evolution are quiescent galaxies with low inferred dispersions, whose number density increases by a factor of ~4 since z=1.5. This build-up leads to an evolution in the quiescent fraction of galaxies such that the threshold dispersion above which quiescent galaxies dominate the counts moves to lower velocity dispersion with time. We show that our results are qualitatively consistent with a simple model in which star-forming galaxies quench and are added to the quiescent population. In order to compensate for the migration into the quiescent population, the velocity dispersions of star-forming galaxies must increase, with a rate that increases with dispersion.