Using a simplified microscopic model of coupled spin and lattice excitations in a ferromagnetic insulator we evaluate the magnetic-field dependence of the spin Seebeck effect at low temperatures. The model includes Heisenberg exchange coupling, a harmonic lattice potential, and a pseudo-dipolar exchange interaction. Our approach goes beyond previous work [Phys. Rev. B 98, 134421 (2018)] in that it does not rely on the a priori assumption of a fast equilibration of the magnon and phonon distributions. Our theory shows that singular features in the magnetic-field dependence of the spin Seebeck effect at low temperatures observed by Kikkawa et al. [Phys. Rev. Lett. 117, 207203 (2016)] are independent of the relative strength of magnon-impurity and phonon-impurity scattering.