Using single crystal inelastic neutron scattering with and without application of an external magnetic field and powder neutron diffraction, we have characterized magnetic interactions in Ba$_3$Cr$_2$O$_8$. Even without field, we found that there exist three singlet-to-triplet excitation modes in $(h,h,l)$ scattering plane. Our complete analysis shows that the three modes are due to spatially anisotropic interdimer interactions that are induced by local distortions of the tetrahedron of oxygens surrounding the Jahn-Teller active Cr$^{5+} (3d^1)$. The strong intradimer coupling of $J_0 = 2.38(2)$ meV and weak interdimer interactions ($|J_{rm inter}| leq 0.52(2)$ meV) makes Ba$_3$Cr$_2$O$_8$ a good model system for weakly-coupled $s = 1/2$ quantum spin dimers.