The thermal conductivity $kappa$ of the Kondo insulator SmB$_6$ was measured at low temperature, down to 70 mK, in magnetic fields up to 15 T, on single crystals grown using both the floating-zone and the flux methods. The residual linear term $kappa_0/T$ at $T to 0$ is found to be zero in all samples, for all magnetic fields, in agreement with previous studies. There is therefore no clear evidence of fermionic heat carriers. In contrast to some prior data, we observe a large enhancement of $kappa(T)$ with increasing field. The effect of field is anisotropic, depending on the relative orientation of field and heat current (parallel or perpendicular), and with respect to the cubic crystal structure. We interpret our data in terms of heat transport predominantly by phonons, which are scattered by magnetic impurities.