We studied the spin depolarization of ensembles of nitrogen-vacancy (NV) centers in nitrogen-rich single crystal diamonds. We found a strong dependence of the evolution of the polarized state in the dark on the concentration of NV centers. At low excitation power, we observed a simple exponential decay profile in the low-density regime and a paradoxical inverted exponential profile in the high-density regime. At higher excitation power, we observed complex behavior, with an initial sharp rise in luminescence signal after the preparation pulse followed by a slower exponential decay. Magnetic field and excitation laser power-dependent measurements suggest that the rapid initial increase of the luminescence signal is related to recharging of the nitrogen-vacancy centers (from neutral to negatively charged) in the dark. The slow relaxing component corresponds to the longitudinal spin relaxation of the NV ensemble. The shape of the decay profile reflects the interplay between two mechanisms: the NV charge state conversion in the dark and the longitudinal spin relaxation. These mechanisms, in turn, are influenced by ionization, recharging and polarization dynamics during excitation. Interestingly, we found that charge dynamics are dominant in NV-dense samples even at very feeble excitation power. These observations may be important for the use of ensembles of NV centers in precession magnetometry and sensing applications.