Recent work has raised the exciting possibility that r-modes (Rossby waves) in rotating neutron star cores might be strong gravitational wave sources. We estimate the effect of a solid crust on their viscous damping rate and show that the dissipation rate in the viscous boundary layer between the oscillating fluid and the nearly static crust is >10^5 times higher than that from the shear throughout the interior. This increases the minimum frequency for the onset of the gravitational r-mode instability to at least 500 Hz when the core temperature is less than 10^10 K. It eliminates the conflict of the r-mode instability with the accretion-driven spin-up scenario for millisecond radio pulsars and makes it unlikely that the r-mode instability is active in accreting neutron stars. For newborn neutron stars, the formation of a solid crust shortly after birth affects their gravitational wave spin-down and hence detectability by ground-based interferometric gravitational wave detectors.