Nonresonant inelastic x-ray scattering (NIXS) has been performed on single crystals of UO$_2$ to study the direction dependence of higher-order-multipole scattering from the uranium $O_{4,5}$ edges (90--110 eV). By comparing the experimental results with theoretical calculations the symmetry of the ground state is confirmed directly as the crystal-field (CF) $Gamma_5$ triplet state within the $J$ = 4 manifold. The results also demonstrate that the directional dichroism of the NIXS spectra is sensitive to the CF strength and establish NIXS as a tool for probing CF interactions quantitatively.