We present a theoretical study of the the effects of off-resonant polarized optical fields on a ferromagnetic model system. We determine the light-induced dynamics of itinerant carriers in a system that includes magnetism at the mean-field level and spin-orbit coupling. We investigate an all-optical switching process for ferromagnets, which is close to the one proposed by Qaiumzadeh et al. [Phys. Rev. B 88, 064416] for the inverse Faraday effect. By computing the optically driven coherent dynamics together with incoherent scattering mechanisms we go beyond a perturbation expansion in powers of the optical field. We find an important contribution of a dynamic Stark effect coupling of the Raman type between the magnetic bands, which leads to a polarization-dependent effect on the magnetization that may support or oppose switching, but also contributes to demagnetization via an increase in electronic energy.