We investigate the interactions of photoexcited carriers with lattice vibrations in thin films of the layered transition metal dichalcogenide (TMDC) WSe$_2$. Employing femtosecond electron diffraction with monocrystalline samples and first principle density functional theory calculations, we obtain a momentum-resolved picture of the energy-transfer from excited electrons to phonons. The measured momentum-dependent phonon population dynamics are compared to first principle calculations of the phonon linewidth and can be rationalized in terms of electronic phase-space arguments. The relaxation of excited states in the conduction band is dominated by intervalley scattering between $Sigma$ valleys and the emission of zone-boundary phonons. Transiently, the momentum-dependent electron-phonon coupling leads to a non-thermal phonon distribution, which, on longer timescales, relaxes to a thermal distribution via electron-phonon and phonon-phonon collisions. Our results constitute a basis for monitoring and predicting out of equilibrium electrical and thermal transport properties for nanoscale applications of TMDCs.