In this paper, we study two-photon interference of broadband chaotic light in a Michelson interferometer with two-photon-absorption detector. The theoretical analysis is based on two-photon interference and Feynman path integral theory. The two-photon coherence matrix is introduced to calculate the second-order interference pattern with polarizations being taken into account. Our study shows that the polarization is another dimension, as well as time and space, to tune the interference pattern in the two-photon interference process. It can act as a switch to manipulate the interference process and open the gate to many new experimental schemes.