The optical transition linewidth and emission polarization of single nitrogen-vacancy (NV) centers are measured from 5 K to room temperature. Inter-excited state population relaxation is shown to broaden the zero-phonon line and both the relaxation and linewidth are found to follow a T^5 dependence for T up to 100 K. This dependence indicates that the dynamic Jahn-Teller effect is the dominant dephasing mechanism for the NV optical transitions at low temperatures.