Recent experiments have measured the signatures of the Kondo effect in the zero-field thermopower of strongly correlated quantum dots [Svilans {em et al.,} Phys. Rev. Lett. {bf 121}, 206801 (2018); Dutta {em et al.,} Nano Lett. {bf 19}, 506 (2019)]. They confirm the predicted Kondo-induced sign change in the thermopower, upon increasing the temperature through a gate-voltage dependent value $T_{1}gtrsim T_{rm K}$, where $T_{rm K}$ is the Kondo temperature. Here, we use the numerical renormalization group (NRG) method to investigate the effect of a finite magnetic field $B$ on the thermopower of such quantum dots. We show that, for fields $B$ exceeding a gate-voltage dependent value $B_{0}$, an additional sign change takes place in the Kondo regime at a temperature $T_{0}(Bgeq B_{0})>0$ with $T_0<T_1$. The field $B_{0}$ is comparable to, but larger than, the field $B_{c}$ at which the zero-temperature spectral function splits in a magnetic field. The validity of the NRG results for $B_{0}$ are checked by comparison with asymptotically exact higher-order Fermi-liquid calculations [Oguri {em et al.,} Phys. Rev. B {bf 97}, 035435 (2018)]. Our calculations clarify the field-dependent signatures of the Kondo effect in the thermopower of Kondo-correlated quantum dots and explain the recently measured trends in the $B$-field dependence of the thermoelectric response of such systems [Svilans {em et al.,} Phys. Rev. Lett. {bf 121}, 206801 (2018)].