In this report, the polarization properties of the photoluminescence emitted by single nitrogen-vacancy (NV) color centers in diamond are investigated using resonant excitation at cryogenic temperature. We first underline that the two excited-state orbital branches are associated with two orthogonal transition dipoles. Using selective excitation of one dipole, we then show that the photoluminescence is partially unpolarized owing to fast relaxation between the two orbitals induced by the thermal bath. This result might be important in the context of the realization of indistinguishable single photons using NV defect in diamond.