Pairs of photons entangled in their time-frequency degree of freedom are of great interest in quantum optics research and applications, due to their relative ease of generation and their high capacity for encoding information. Here we analyze, both theoretically and experimentally, the behavior of phase-insensitive spectrally-resolved interferences arising from two pairs of time-frequency entangled photons. At its core, this is a multimode entanglement swapping experiment, whereby a spectrally resolved joint measurement on the idler photons from both pairs results in projecting the signal photons onto a Bell state whose form depends on the measurement outcome. Our analysis is a thorough exploration of what can be achieved using time-frequency entanglement and spectrally-resolved Bell-state measurements.