Plasmons are usually described in terms of macroscopic quantities such as electric fields and currents. However as fundamental excitations of metals they are also quantum objects with internal structure. We demonstrate that this can induce an intrinsic dipole moment which is tied to the quantum geometry of the Hilbert space of plasmon states. This {it quantum geometric dipole} offers a unique handle for manipulation of plasmon dynamics, via density modulations and electric fields. As a concrete example we demonstrate that scattering of plasmons with non-vanishing quantum geometric dipole from impurities is non-reciprocal, skewing in different directions in a valley-dependent fashion. This internal structure can be used to control plasmon trajectories in two dimensional materials.