Obtaining coarse-grained models that accurately incorporate finite-size effects is an important open challenge in the study of complex, multi-scale systems. We apply Langevin regression, a recently developed method for finding stochastic differential equation (SDE) descriptions of realistically-sampled time series data, to understand finite-size effects in the Kuramoto model of coupled oscillators. We find that across the entire bifurcation diagram, the dynamics of the Kuramoto order parameter are statistically consistent with an SDE whose drift term has the form predicted by the Ott-Antonsen ansatz in the $Nto infty$ limit. We find that the diffusion term is nearly independent of the bifurcation parameter, and has a magnitude decaying as $N^{-1/2}$, consistent with the central limit theorem. This shows that the diverging fluctuations of the order parameter near the critical point are driven by a bifurcation in the underlying drift term, rather than increased stochastic forcing.