Levitated optomechanical systems are rapidly becoming leading tools for precision sensing, enabling a high level of control over the sensors center of mass motion, rotation and electric charge state. Higher-order multipole moments in the charge distribution, however, remain a major source of backgrounds. By applying controlled precessive torques to the dipole moment of a levitated microsphere in vacuum, we demonstrate cancellation of dipole-induced backgrounds by 2 orders of magnitude. We measure the dipole moments of ng-mass spheres and determine their scaling with sphere size, finding that the dominant torques arise from induced dipole moments related to dielectric-loss properties of the SiO$_2$ spheres. Control of multipole moments in the charge distribution of levitated sensors is a key requirement to sufficiently reduce background sources in future applications.