Kinetic simulations and theory demonstrate that whistler waves can excite oblique, short-wavelength fluctuations through secondary drift instabilities if a population of sufficiently cold plasma is present. The excited modes lead to heating of the cold populations and damping of the primary whistler waves. The instability threshold depends on the density and temperature of the cold population and can be relatively small if the temperature of the cold population is sufficiently low. This mechanism may thus play a significant role in controlling amplitude of whistlers in the regions of the Earths magnetosphere where cold background plasma of sufficient density is present.