$Fe_3O_4$ is a mixed-valence strongly correlated transition metal oxide which displays the intriguing metal to insulator Verwey transition. Here we investigate the electronic and magnetic structure of $Fe_3O_4$ by a unique combination of high-resolution Fe 2p3d resonant inelastic scattering magnetic circular (RIXS-MCD) and magnetic linear (RIXS-MLD) dichroism. We show that by coupling the site selectivity of RIXS with the magnetic selectivity imposed by the incident polarization handedness, we can unambiguously identify spin-flip excitations and quantify the exchange interaction of the different sublattices. Furthermore, our RIXS-MLD measurements show spin-orbital excitations that exhibit strong polarization and magnetic field dependence. Guided by theoretical simulations, we reveal that the angular dependence arises from a strong interplay between trigonal crystal-field, magnetic exchange and spin-orbit interaction at the nominal $Fe^{2+}$ sites. Our results highlight the capabilities of RIXS magnetic dichroism studies to investigate the ground state of complex systems where in-equivalent sites and bonds are simultaneously present.