We use radial quantization to compute Chern-Simons partition functions on handlebodies of arbitrary genus. The partition function is given by a particular transition amplitude between two states which are defined on the Riemann surfaces that define the (singular) foliation of the handlebody. The final state is a coherent state while on the initial state the holonomy operator has zero eigenvalue. The latter choice encodes the constraint that the gauge fields must be regular everywhere inside the handlebody. By requiring that the only singularities of the gauge field inside the handlebody must be compatible with Wilson loop insertions, we find that the Wilson loop shifts the holonomy of the initial state. Together with an appropriate choice of normalization, this procedure selects a unique state in the Hilbert space obtained from a Kahler quantization of the theory on the constant-radius Riemann surfaces. Radial quantization allows us to find the partition functions of Abelian Chern-Simons theories for handlebodies of arbitrary genus. For non-Abelian compact gauge groups, we show that our method reproduces the known partition function at genus one.