Observations have demonstrated that supernovae efficiently produce dust. This is consistent with the hypothesis that supernovae and asymptotic giant branch stars are the primary producers of dust in the Universe. However, there has been a longstanding question of how much of the dust detected in the interiors of young supernova remnants can escape into the interstellar medium. We present new hydrodynamical calculations of the evolution of dust grains that were formed in dense ejecta clumps within a Cas A-like remnant. We follow the dynamics of the grains as they decouple from the gas after their clump is hit by the reverse shock. They are subsequently subject to destruction by thermal and kinetic sputtering as they traverse the remnant. Grains that are large enough ($sim 0.25,mu$m for silicates and $sim 0.1,mu$m for carbonaceous grains) escape into the interstellar medium while smaller grains get trapped and destroyed. However, grains that reach the interstellar medium still have high velocities, and are subject to further destruction as they are slowed down. We find that for initial grain size distributions that include large ($sim 0.25 - 0.5,mu$m) grains, 10--20% of silicate grains can survive, while 30--50% of carbonaceous grains survive even when the initial size distribution cuts off at smaller ($0.25,mu$m) sizes. For a 19 M$_{odot}$ star similar to the progenitor of Cas A, up to 0.1 M$_{odot}$ of dust can survive if the dust grains formed are large. Thus we show that supernovae under the right conditions can be significant sources of interstellar dust.