While cubic Quasi-topological gravity is unique, there is a family of quartic Quasi-topological gravities in five dimensions. These theories are defined by leading to a first order equation on spherically symmetric spacetimes, resembling the structure of the equations of Lovelock theories in higher-dimensions, and are also ghost free around AdS. Here we construct slowly rotating black holes in these theories, and show that the equations for the off-diagonal components of the metric in the cubic theory are automatically of second order, while imposing this as a restriction on the quartic theories allows to partially remove the degeneracy of these theories, leading to a three-parameter family of Lagrangians of order four in the Riemann tensor. This shows that the parallel with Lovelock theory observed on spherical symmetry, extends to the realm of slowly rotating solutions. In the quartic case, the equations for the slowly rotating black hole are obtained from a consistent, reduced action principle. These functions admit a simple integration in terms of quadratures. Finally, in order to go beyond the slowly rotating regime, we explore the consistency of the Kerr-Schild ansatz in cubic Quasi-topological gravity. Requiring the spacetime to asymptotically match with the rotating black hole in GR, for equal oblateness parameters, the Kerr-Schild deformation of an AdS vacuum, does not lead to a solution for generic values of the coupling. This result suggests that in order to have solutions with finite rotation in Quasi-topological gravity, one must go beyond the Kerr-Schild ansatz.