We revisit the spectrum of pure quantum gravity in AdS$_3$. The computation of the torus partition function will -- if computed using a gravitational path integral that includes only smooth saddle points -- lead to a density of states which is not physically sensible, as it has a negative degeneracy of states for some energies and spins. We consider a minimal cure for this non-unitarity of the pure gravity partition function, which involves the inclusion of additional states below the black hole threshold. We propose a geometric interpretation for these extra states: they are conical defects with deficit angle $2pi(1-1/N)$, where $N$ is a positive integer. That only integer values of $N$ should be included can be seen from a modular bootstrap argument, and leads us to propose a modest extension of the set of saddle-point configurations that contribute to the gravitational path integral: one should sum over orbifolds in addition to smooth manifolds. These orbifold states are below the black hole threshold and are regarded as massive particles in AdS, but they are not perturbative states: they are too heavy to form multi-particle bound states. We compute the one-loop determinant for gravitons in these orbifold backgrounds, which confirms that the orbifold states are Virasoro primaries. We compute the gravitational partition function including the sum over these orbifolds and find a finite, modular invariant result; this finiteness involves a delicate cancellation between the infinite tower of orbifold states and an infinite number of instantons associated with $PSL(2,{mathbb Z})$ images.