Sunspot number series are subject to various uncertainties, which are still poorly known. The need for their better understanding was recently highlighted by the major makeover of the international Sunspot Number [Clette et al., Space Science Reviews, 2014]. We present the first thorough estimation of these uncertainties, which behave as Poisson-like random variables with a multiplicative coefficient that is time- and observatory-dependent. We provide a simple expression for these uncertainties, and reveal how their evolution in time coincides with changes in the observations, and processing of the data. Knowing their value is essential for properly building composites out of multiple observations, and for preserving the stability of the composites in time.