We study the spectrum of pure massless higher spin theories in $AdS_3$. The light spectrum is given by a tower of massless particles of spin $s=2,cdots,N$ and their multi-particles states. Their contribution to the torus partition function organises into the vacuum character of the ${cal W}_N$ algebra. Modular invariance puts constraints on the heavy spectrum of the theory, and in particular leads to negative norm states, which would be inconsistent with unitarity. This negativity can be cured by including additional light states, below the black hole threshold but whose mass grows with the central charge. We show that these states can be interpreted as conical defects with deficit angle $2pi(1-1/M)$. Unitarity allows the inclusion of such defects into the path integral provided $M geq N$.