Van der Waals semiconductor heterostructures could be a platform to harness hot photoexcited carriers in the next generation of optoelectronic and photovoltaic devices. The internal quantum efficiency of hot-carrier devices is determined by the relation between photocarrier extraction and thermalization rates. Using textit{ab-initio} methods we show that the photocarrier thermalization time in single-layer transition metal dichalcogenides strongly depends on the peculiarities of the phonon spectrum and the electronic spin-orbit coupling. In detail, the lifted spin degeneracy in the valence band suppresses the hole scattering on acoustic phonons, slowing down the thermalization of holes by one order of magnitude as compared to electrons. Moreover, the hole thermalization time behaves differently in MoS$_2$ and WSe$_2$ because spin-orbit interactions differ in these seemingly similar materials. We predict that the internal quantum efficiency of a tunneling van der Waals semiconductor heterostructure depends qualitatively on whether MoS$_2$ or WSe$_2$ is used.