We demonstrate a method to reduce number fluctuations in an ultracold atomic sample using real-time feedback. By measuring the Faraday rotation of an off-resonant probe laser beam with a pair of avalanche photodetectors in a polarimetric setup we produce a proxy for the number of atoms in the sample. We iteratively remove a fraction of the excess atoms from the sample to converge on a target proxy value in a way that is insensitive to environmental perturbations and robust to errors in light polarization. Using absorption imaging for out-of-loop verification, we demonstrate a reduction in the number fluctuations from $3%$ to $0.45%$ for samples at a temperature of 16.4 $mu$K over the time-scale of several hours which is limited by temperature fluctuations, beam pointing noise, and photon shot noise.