We analyze state preparation within a restricted space of local control parameters between adiabatically connected states of control Hamiltonians. We formulate a conjecture that the time integral of energy fluctuations over the protocol duration is bounded from below by the geodesic length set by the quantum geometric tensor. The conjecture implies a geometric lower bound for the quantum speed limit (QSL). We prove the conjecture for arbitrary, sufficiently slow protocols using adiabatic perturbation theory and show that the bound is saturated by geodesic protocols, which keep the energy variance constant along the trajectory. Our conjecture implies that any optimal unit-fidelity protocol, even those that drive the system far from equilibrium, are fundamentally constrained by the quantum geometry of adiabatic evolution. When the control space includes all possible couplings, spanning the full Hilbert space, we recover the well-known Mandelstam-Tamm bound. However, using only accessible local controls to anneal in complex models such as glasses or to target individual excited states in quantum chaotic systems, the geometric bound for the quantum speed limit can be exponentially large in the system size due to a diverging geodesic length. We validate our conjecture both analytically by constructing counter-diabatic and fast-forward protocols for a three-level system, and numerically in nonintegrable spin chains and a nonlocal SYK model.