We investigated the ground state symmetry of the cubic hidden order compound CeB$_6$ by means of core level non-resonant inelastic x-ray scattering (NIXS). The information is obtained from the directional dependence of the scattering function that arises from higher than dipole transitions. Our new method confirms that the ground state is well described using a localized crystal-field model assuming a $Gamma_8$ quartet ground state.