We construct the scalar potential for the exceptional field theory based on the affine symmetry group E$_9$. The fields appearing in this potential live formally on an infinite-dimensional extended spacetime and transform under E$_9$ generalised diffeomorphisms. In addition to the scalar fields expected from D=2 maximal supergravity, the invariance of the potential requires the introduction of new constrained scalar fields. Other essential ingredients in the construction include the Virasoro algebra and indecomposable representations of E$_9$. Upon solving the section constraint, the potential reproduces the dynamics of either eleven-dimensional or type IIB supergravity in the presence of two isometries.