We report photometric and radial velocity observations of the XO-4 transiting planetary system, conducted with the FLWO 1.2m telescope and the 8.2m Subaru Telescope. Based on the new light curves, the refined transit ephemeris of XO-4b is $P = 4.1250828 pm 0.0000040$ days and $T_c [BJD_TDB] = 2454485.93323 pm 0.00039$. We measured the Rossiter-McLaughlin effect of XO-4b and estimated the sky-projected angle between the stellar spin axis and the planetary orbital axis to be $lambda = -46.7^{circ} ^{+8.1^{circ}}_{-6.1^{circ}}$. This measurement of $lambda$ is less robust than in some other cases because the impact parameter of the transit is small, causing a strong degeneracy between $lambda$ and the projected stellar rotational velocity. Nevertheless, our finding of a spin-orbit misalignment suggests that the migration process for XO-4b involved few-body dynamics rather than interaction with a gaseous disk. In addition, our result conforms with the pattern reported by Winn et al. (2010, ApJL, 718, L145) that high obliquities are preferentially found for stars with effective temperatures hotter than 6250~K.