We have investigated the temperature evolution of the magnetic structures of HoFeO$_3$ by single crystal neutron diffraction. The three different magnetic structures found as a function of temperature for hfo are described by the magnetic groups Pb$$n$2_1$, Pbn$2_1$ and Pbn$2_1$ and are stable in the temperature ranges $approx$ 600-55~K, 55-37~K and 35$>T>2$~K respectively. In all three the fundamental coupling between the Fe sub-lattices remains the same and only their orientation and the degree of canting away from the ideal axial direction varies. The magnetic polarisation of the Ho sub-lattices in these two higher temperature regions, in which the major components of the Fe moment lie along $x$ and $y$, is very small. The canting of the moments from the axial directions is attributed to the antisymmetric interactions allowed by the crystal symmetry. They include contributions from single ion anisotropy as well as the Dzyaloshinski antisymmetric exchange. In the low temperature phase two further structural transitions are apparent in which the spontaneous magnetisation changes sign with respect to the underlying antiferromagnetic configuration. In this temperature range the antisymmetric exchange energy varies rapidly as the the Ho sub-lattices begin to order. So long as the ordered Ho moments are small the antisymmetric exchange is due only to Fe-Fe interactions, but as the degree of Ho order increases the Fe-Ho interactions take over whilst at the lowest temperatures, when the Ho moments approach saturation the Ho-Ho interactions dominate. The reversals of the spontaneous magnetisation found in this study suggest that in hfo the sums of the Fe-Fe and Ho-Ho antisymmetric interactions have the same sign as one another, but that of the Ho-Fe terms is opposite.