Lateral Casimir force near a laterally-inhomogeneous plate is first revealed by both rigorous simulations and proximity approximations. The inhomogeneity-induced lateral Casimir force provides a novel method to control the lateral motion of nano-objects above the plate, and makes source-free manipulations of them possible. When incorporated with the Casimir repulsion in a fluid, the lateral Casimir force is shown to dominate over Brownian motion and enables long-distance quantum propulsion and firm quantum trapping of nano-objects. Gratings of varying filling factors to mimic micro-scale inhomogeneity also confirm those effects. The idea to design asymmetric distributions of nano-structures paves the way to sophisticated tailoring of the lateral Casimir force.