Si dominates the semiconductor industry material but possesses an abnormally low room temperature hole mobility (505 cm^2/Vs), which is four times lower than that of Diamond and Ge (2000 cm^2/Vs), two adjacent neighbours in the group IV column in the Periodic Table. In the past half-century, extensive efforts have been made to overcome the challenges of Si technology caused by low mobility in Si. However, the fundamental understanding of the underlying mechanisms remains lacking. Here, we theoretically reproduce the experimental data for conventional group IV and III-V semiconductors without involving adjustable parameters by curing the shortcoming of classical models. We uncover that the abnormally low hole mobility in Si originating from a combination of the strong interband scattering resulting from its weak spin-orbit coupling and the intensive participation of optical phonons in hole-phonon scattering. In contrast, the strong spin-orbit coupling in Ge leads to a negligible interband scattering; the strong bond and light atom mass in diamond give rise to high optical phonons frequency, preventing their participation in scattering. Based on these understandings rooted into the fundamental atomic properties, we present design principles for semiconducting materials towards high hole mobility.