Growing out of the initial connections between subfactors and knot theory that gave rise to the Jones polynomial, Jones axiomatization of the standard invariant of an extremal finite index $II_1$ subfactor as a spherical $C^*$-planar algebra, presented in arXiv:math.QA/9909027, is the most elegant and powerful description available. We make the natural extension of this axiomatization to the case of finite index subfactors of arbitrary type. We also provide the first steps toward a limited planar structure in the infinite index case. The central role of rotations, which provide the main non-trivial part of the planar structure, is a recurring theme throughout this work. In the finite index case the axioms of a $C^*$-planar algebra need to be weakened to disallow rotation of internal discs, giving rise to the notion of a rigid $C^*$-planar algebra. We show that the standard invariant of any finite index subfactor has a rigid $C^*$-planar algebra structure. We then show that rotations can be re-introduced with associated correction terms entirely controlled by the Radon-Nikodym derivative of the two canonical states on the first relative commutant, $N cap M$. By deforming a rigid $C^*$-planar algebra to obtain a spherical $C^*$-planar algebra and lifting the inverse construction to the subfactor level we show that any rigid $C^*$-planar algebra arises as the standard invariant of a finite index $II_1$ subfactor equipped with a conditional expectation, which in general is not trace preserving. Jones results thus extend completely to the general finite index case. We conclude by applying our machinery to the $II_1$ case, shedding new light on the rotations studied by Huang [11] and touching briefly on the work of Popa [29]. (continued in article)