The time-frequency structure of quantum light can be manipulated for information processing and metrology. Characterizing this structure is also important for developing quantum light sources with high modal purity that can interfere with other independent sources. Here, we present and experimentally demonstrate a scheme based on intensity interferometry to measure the joint spectral mode of photon pairs produced by spontaneous parametric down-conversion. We observe correlations in the spectral phase of the photons due to chirp in the pump. We also show that our scheme can be combined with stimulated emission tomography to quickly measure their mode using bright classical light. Our scheme does not require phase stability, nonlinearities, or spectral shaping, and thus is an experimentally simple way of measuring the modal structure of quantum light.