Modular invariance strongly constrains the spectrum of states of two dimensional conformal field theories. By summing over the images of the modular group, we construct candidate CFT partition functions that are modular invariant and have positive spectrum. This allows us to efficiently extract the constraints on the CFT spectrum imposed by modular invariance, giving information on the spectrum that goes beyond the Cardy growth of the asymptotic density of states. Some of the candidate modular invariant partition functions we construct have gaps of size (c-1)/12, proving that gaps of this size and smaller are consistent with modular invariance. We also revisit the partition function of pure Einstein gravity in AdS3 obtained by summing over geometries, which has a spectrum with two unphysical features: it is continuous, and the density of states is not positive definite. We show that both of these can be resolved by adding corrections to the spectrum which are subleading in the semi-classical (large central charge) limit.