Electro-optic modulators are utilized ubiquitously ranging from applications in data communication to photonic neural networks. While tremendous progress has been made over the years, efficient phase-shifting modulators are challenged with fundamental tradeoffs, such as voltage-length, index change-losses or energy-bandwidth, and no single solution available checks all boxes. While voltage-driven phase modulators, such as based on lithium niobate, offer low loss and high speed operation, their footprint of 10s of cm-scale is prohibitively large, especially for density-critical applications, for example in photonic neural networks. Ignoring modulators for quantum applications, where loss is critical, here we distinguish between current versus voltage-driven modulators. We focus on the former, since current-based schemes of emerging thin electro-optical materials have shown unity-strong index modulation suitable for heterogeneous integration into foundry waveguides. Here, we provide an in-depth ab-initio analysis of obtainable modulator performance based on heterogeneously integrating low-dimensional materials, i.e. graphene, thin films of indium tin oxide, and transition metal dichalcogenide monolayers into a plurality of optical waveguide designs atop silicon photonics. Using the fundamental modulator tradeoff of energy-bandwidth-product as a design-quality quantifier, we show that a small modal cross section, such as given by plasmonic modes, enables high-performance operation, physically realized by arguments on charge-distribution and low electrical resistance. An in-depth design understanding of phase-modulator performance, beyond doped-junctions in silicon, offers opportunities for micrometer-compact yet energy-bandwidth-ratio constrained modulators with timely opportunities to hardware-accelerate applications beyond data communication towards photonic machine intelligence.