We investigate quaternionic contact (qc) manifolds from the point of view of intrinsic torsion. We argue that the natural structure group for this geometry is a non-compact Lie group K containing Sp(n)H^*, and show that any qc structure gives rise to a canonical K-structure with constant intrinsic torsion, except in seven dimensions, when this condition is equivalent to integrability in the sense of Duchemin. We prove that the choice of a reduction to Sp(n)H^* (or equivalently, a complement of the qc distribution) yields a unique K-connection satisfying natural conditions on torsion and curvature. We show that the choice of a compatible metric on the qc distribution determines a canonical reduction to Sp(n)Sp(1) and a canonical Sp(n)Sp(1)-connection whose curvature is almost entirely determined by its torsion. We show that its Ricci tensor, as well as the Ricci tensor of the Biquard connection, has an interpretation in terms of intrinsic torsion.