We show that the phase space of three-dimensional gravity contains two layers of dualities: between diffeomorphisms and a notion of dual diffeomorphisms on the one hand, and between first order curvature and torsion on the other hand. This is most elegantly revealed and understood when studying the most general Lorentz-invariant first order theory in connection and triad variables, described by the so-called Mielke-Baekler Lagrangian. By analyzing the quasi-local symmetries of this theory in the covariant phase space formalism, we show that in each sector of the torsion/curvature duality there exists a well-defined notion of dual diffeomorphism, which furthermore follows uniquely from the Sugawara construction. Together with the usual diffeomorphisms, these duals form at finite distance, without any boundary conditions, and for any sign of the cosmological constant, a centreless double Virasoro algebra which in the flat case reduces to the BMS$_3$ algebra. These algebras can then be centrally-extended via the twisted Sugawara construction. This shows that the celebrated results about asymptotic symmetry algebras are actually generic features of three-dimensional gravity at any finite distance. They are however only revealed when working in first order connection and triad variables, and a priori inaccessible from Chern-Simons theory. As a bonus, we study the second order equations of motion of the Mielke-Baekler model, as well as the on-shell Lagrangian. This reveals the duality between Riemannian metric and teleparallel gravity, and a new candidate theory for three-dimensional massive gravity which we call teleparallel topologically massive gravity.