We investigated the crystal-electric field ground state of the 4$f$ manifold in the strongly correlated topological insulator SmB$_6$ using core level non-resonant inelastic x-ray scattering (NIXS). The directional dependence of the scattering function that arises from higher multipole transitions establishes unambiguously that the $Gamma_8$ quartet state of the Sm $f^5$ $J$=$5/2$ configuration governs the ground-state symmetry and hence the topological properties of SmB$_6$. Our findings contradict the results of density functional calculations reported so far.