We present a study of the charge state conversion of single nitrogen-vacancy (NV) defects hosted in nanodiamonds (NDs). We first show that the proportion of negatively-charged NV$^{-}$ defects, with respect to its neutral counterpart NV$^{0}$, decreases with the size of the ND. We then propose a simple model based on a layer of electron traps located at the ND surface which is in good agreement with the recorded statistics. By using thermal oxidation to remove the shell of amorphous carbon around the NDs, we demonstrate a significant increase of the proportion of NV$^{-}$ defects in 10-nm NDs. These results are invaluable for further understanding, control and use of the unique properties of negatively-charged NV defects in diamond