Magnetostatic modes supported by a ferromagnetic sphere have been known as the Walker modes, each of which possesses an orbital angular momentum as well as a spin angular momentum along a static magnetic field. The Walker modes with non-zero orbital angular momenta exhibit topologically non-trivial spin textures, which we call textit{magnetic quasi-vortices}. Photons in optical whispering gallery modes supported by a dielectric sphere possess orbital and spin angular momenta forming textit{optical vortices}. Within a ferromagnetic, as well as dielectric, sphere, two forms of vortices interact in the process of Brillouin light scattering. We argue that in the scattering there is a selection rule that dictates the exchange of orbital angular momenta between the vortices. The selection rule is shown to be responsible for the experimentally observed nonreciprocal Brillouin light scattering.