We investigate ultrafast demagnetization due to electron-phonon interaction in a model band-ferromagnet. We show that the microscopic mechanism behind the spin dynamics due to electron-phonon interaction is the interplay of scattering and the precession around momentum-dependent effective internal spin-orbit magnetic fields. The resulting magnetization dynamics can only be mimicked by spin-flip transitions if the spin precession around the internal fields is sufficiently fast (compared to the scattering time) so that it averages out the transverse spin components.