We argue that impact velocities between dust grains with sizes less than $sim 0.1$ $mu m$ in molecular cloud cores are dominated by drift arising from ambipolar diffusion. This effect is due to the size dependence of the dust coupling to the magnetic field and the neutral gas. Assuming perfect sticking in collisions up to $approx 50$ m/s, we show that this effect causes rapid depletion of small grains - consistent with starlight extinction and IR/microwave emission measurements, both in the core center ($n sim 10^{6}$ cm$^{-3}$) and envelope ($n sim 10^{4}$ cm$^{-3}$). The upper end of the size distribution does not change significantly if only velocities arising from this effect are considered. We consider the impact of an evolved dust size distribution on the gas temperature, and argue that if the depletion of small dust grains occurs as would be expected from our model, then the cosmic ray ionization rate must be well below $10^{-16}$ s$^{-1}$ at a number density of $10^{5}$ cm$^{-3}$.