The two-photon Hong-Ou-Mandel (HOM) interference is a pure quantum effect which indicates the degree of indistinguishability of photons. The four-photon HOM interference exhibits richer dynamics in comparison to the two-photon interference and simultaneously is more sensitive to the input photon states. We demonstrate theoretically and experimentally an explicit dependency of the four-photon interference to the number of temporal modes, created in the process of parametric down-conversion. Moreover, we exploit the splitting ratio of the beam splitter to manipulate the interference between bunching and antibunching. Our results reveal that the temporal mode structure (multimodeness) of the quantum states shapes many-particle interference.