Observations show various jets in the solar atmosphere with significant rotational motions, which may undergo instabilities leading to heat ambient plasma. We study the Kelvin-Helmholtz (KH) instability of twisted and rotating jets caused by the velocity jumps near the jet surface. We derive a dispersion equation with appropriate boundary condition for total pressure (including centrifugal force of tube rotation), which governs the dynamics of incompressible jets. Then, we obtain analytical instability criteria of Kelvin-Helmholtz instability in various cases, which were verified by numerical solutions to the dispersion equation. We find that twisted and rotating jets are unstable to KH instability when the kinetic energy of rotation is more than the magnetic energy of the twist. Our analysis shows that the azimuthal magnetic field of 1-5 G can stabilize observed rotations in spicule/macrospicules and X-ray/EUV jets. On the other hand, non-twisted jets are always unstable to KH instability. In this case, the instability growth time is several seconds for spicule/macrospicules and few minutes (or less) for EUV/X-ray jets. We also find that standing kink and torsional Alfven waves are always unstable near the antinodes due to the jump of azimuthal velocity at the surface, while the propagating waves are generally stable. KH vortices may lead to enhanced turbulence development and heating of surrounding plasma, therefore rotating jets may provide energy for chromospheric and coronal heating.