We show that an isotropic dipolar particle in the vicinity of a substrate made of nonreciprocal plasmonic materials can experience a lateral Casimir force and torque when the particles temperature differs from that of the slab and the environment. We connect the existence of the lateral force to the asymmetric dispersion of nonreciprocal surface polaritons and the existence of the lateral torque to the spin-momentum locking of such surface waves. Using the formalism of fluctuational electrodynamics, we show that the features of lateral force and torque should be experimentally observable using a substrate of doped Indium Antimonide (InSb) placed in an external magnetic field, and for a variety of dielectric particles. Interestingly, we also find that the directions of the lateral force and the torque depend on the constituent materials of the particles, which suggests a sorting mechanism based on lateral nonequilibrium Casimir physics.