We investigate the thermal quenching of the multimodal photoluminescence from InAs/InP (001) self-assembled quantum dots. The temperature evolution of the photoluminescence spectra of two samples is followed from 10 K to 300 K. We develop a coupled rate-equation model that includes the effect of carrier thermal escape from a quantum dot to the wetting layer and to the InP matrix, followed by transport, recapture or non-radiative recombination. Our model reproduces the temperature dependence of the emission of each family of quantum dots with a single set of parameters. We find that the main escape mechanism of the carriers confined in the quantum dots is through thermal emission to the wetting layer. The activation energy for this process is found to be close to one-half the energy difference between that of a given family of quantum dots and that of the wetting layer as measured by photoluminescence excitation experiments. This indicates that electron and holes exit the InAs quantum dots as correlated pairs.