We present VLT/SINFONI imaging spectroscopy of the warm ionized gas in 33 powerful radio galaxies at redshifts z>~2, which are excellent sites to study the interplay of rapidly accreting active galactic nuclei and the interstellar medium of the host galaxy in the very late formation stages of massive galaxies. Our targets span two orders of magnitude in radio size (2-400 kpc) and kinetic jet energy (a few 10^46 to almost 10^48 erg s^-1). All sources have complex gas kinematics with broad line widths up to ~1300 km s^-1. About half have bipolar velocity fields with offsets up to 1500 km s^-1 and are consistent with global back-to-back outflows. The others have complex velocity distributions, often with multiple abrupt velocity jumps far from the nucleus of the galaxy, and are not associated with a major merger in any obvious way. We present several empirical constraints that show why gas kinematics and radio jets seem to be physically related. The gas kinetic energy from large scale bulk and local outflow or turbulent motion corresponds to a few 10^-3 to 10^-2 of the kinetic energy of the jet. In galaxies with jet power >~10^47 erg s^-1, the bulk kinetic energy dominates the total energy budget of the gas, suggesting that the outflows encompasses the global interstellar medium, perhaps facilitated by the strong gas turbulence. We compare with recent hydrodynamic simulations, and discuss the potential consequences for the subsequent evolution of massive high-z galaxies. The gas-phase metallicities in our galaxies are lower than in most low-z AGN, but nonetheless solar or even super-solar, suggesting that the ISM in these galaxies is very similar to the gas from which massive low-redshift galaxies formed most of their gas. This further highlights that we are seeing these galaxies near the end of their active formation phase.