We describe new boundary conditions for AdS$_2$ in Jackiw-Teitelboim gravity. The asymptotic symmetry group is enhanced to $r{Diff}(S^1)ltimes C^infty(S^1)$ whose breaking to $r{SL}(2,R)times r{U}(1)$ controls the near-AdS$_2$ dynamics. The action reduces to a boundary term which is a generalization of the Schwarzian theory and can be interpreted as the coadjoint action of the warped Virasoro group. This theory reproduces the low-energy effective action of the complex SYK model. We compute the Euclidean path integral and derive its relation to the random matrix ensemble of Saad, Shenker and Stanford. We study the flat space version of this action, and show that the corresponding path integral also gives an ensemble average, but of a much simpler nature. We explore some applications to near-extremal black holes.