In black hole X-ray binaries, a misalignment between the spin axis of the black hole and the orbital angular momentum can occur during the supernova explosion that forms the compact object. In this letter we present population synthesis models of Galactic black hole X-ray binaries, and study the probability density function of the misalignment angle, and its dependence on our model parameters. In our modeling, we also take into account the evolution of misalignment angle due to accretion of material onto the black hole during the X-ray binary phase. The major factor that sets the misalignment angle for X-ray binaries is the natal kick that the black hole may receive at its formation. However, large kicks tend to disrupt binaries, while small kicks allow the formation of XRBs and naturally select systems with small misalignment angles. Our calculations predict that the majority (>67%) of Galactic field BH XRBs have rather small (>10 degrees) misalignment angles, while some systems may reach misalignment angles as high as ~90 degrees and even higher. This results is robust among all population synthesis models. The assumption of small small misalignment angles is extensively used to observationally estimate black hole spin magnitudes, and for the first time we are able to confirm this assumption using detailed population synthesis calculations.