We present systematic measurements of longitudinal relaxation rates ($1/T_1$) of spin polarization in the ground state of the nitrogen-vacancy (NV$^-$) color center in synthetic diamond as a function of NV$^-$ concentration and magnetic field $B$. NV$^-$ centers were created by irradiating a Type 1b single-crystal diamond along the [100] axis with 200 keV electrons from a transmission electron microscope with varying doses to achieve spots of different NV$^-$ center concentrations. Values of ($1/T_1$) were measured for each spot as a function of $B$.