Temperature plays an important role in spin torque switching of magnetic tunnel junctions causing magnetization fluctuations that decrease the switching voltage but also introduce switching errors. Here we present a systematic study of the temperature dependence of the spin torque switching probability of state-of-the-art perpendicular magnetic tunnel junction nanopillars (40 to 60 nm in diameter) from room temperature down to 4 K, sampling up to a million switching events. The junction temperature at the switching voltage---obtained from the thermally assisted spin torque switching model---saturates at temperatures below about 75 K, showing that junction heating is significant below this temperature and that spin torque switching remains highly stochastic down to 4 K. A model of heat flow in a nanopillar junction shows this effect is associated with the reduced thermal conductivity and heat capacity of the metals in the junction.