Dirac matters provide a platform for exploring the interplay of their carriers with other quantum phenomena. Sr$_{1-x}$Mn$_{1-y}$Sb$_2$ has been proposed to be a magnetic Weyl semimetal and provides an excellent platform to study the coupling between Weyl fermions and magnons. Here, we report comprehensive inelastic neutron scattering (INS) measurements on single crystals of Sr$_{1-x}$Mn$_{1-y}$Sb$_2$, which have been well characterized by magnetization and magnetotransport measurements, both of which demonstrate that the material is a topologically nontrivial semimetal. The INS spectra clearly show a spin gap of $sim6$ meV. The dispersion in the magnetic Mn layer extends up to about 76 meV, while that between the layers has a narrow band width of 6 meV. We find that the linear spin-wave theory using a Heisenberg spin Hamiltonian can reproduce the experimental spectra with the following parameters: a nearest-neighbor ($SJ_1sim28.0$ meV) and next-nearest-neighbor in-plane exchange interaction ($SJ_2sim9.3$ meV) , interlayer exchange coupling ($SJ_csim-0.1$ meV), and spin anisotropy constant ($SDsim-0.07$ meV). Despite the coexistence of Weyl fermions and magnons, we find no clear evidence that the magnetic dynamics are influenced by the Weyl fermions in Sr$_{1-x}$Mn$_{1-y}$Sb$_2$, possibly because that the Weyl fermions and magnons reside in the Sb and Mn layers separately, and the interlayer coupling is weak due to the quasi-two-dimensional nature of the material, as also evident from the small $SJ_c$ of -0.1 meV.