We derive a simple formula for the fluctuations of the time average around the thermal mean for overdamped Brownian motion in a binding potential U(x). Using a backward Fokker-Planck equation, introduced by Szabo, et al. in the context of reaction kinetics, we show that for ergodic processes these finite measurement time fluctuations are determined by the Boltzmann measure. For the widely applicable logarithmic potential, ergodicity is broken. We quantify the large non-ergodic fluctuations and show how they are related to a super-aging correlation function.