Surface plasmon polaritons carrying orbital angular momentum are of great fundamental and applied interest. However, common approaches for their generation are restricted to having a weak dependence on the properties of the plasmon-generating illumination, providing a limited degree of control over the amount of delivered orbital angular momentum. Here we experimentally show that by tailoring local and global geometries of vortex generators, a change in circular polarization handedness of light imposes arbitrary large switching in the delivered plasmonic angular momentum. Using time-resolved photoemission electron microscopy we demonstrate pristine control over the generation and rotation direction of high-order plasmonic vortices. We generalize our approach to create complex topological fields and exemplify it by studying and controlling a bright vortex, exhibiting the breakdown of a high-order vortex into a mosaic of unity-order vortices while maintaining the overall angular momentum density. Our results provide tools for plasmonic manipulation and could be utilized in lab-on-a-chip devices.