The dephasing mechanisms of p-shell and s-shell excitons in an InAs self-assembled quantum dot ensemble are examined using two-dimensional coherent spectroscopy (2DCS). 2DCS provides a comprehensive picture of how the energy level structure of dots affects the exciton dephasing rates and recombination lifetimes. We find that at low temperatures, dephasing of s-shell excitons is lifetime limited, whereas p-shell excitons exhibit significant pure dephasing due to scattering between degenerate spin states. At elevated temperatures, quadratic exciton-phonon coupling plays an important role in both s-shell and p-shell exciton dephasing. We show that multiple p-shell states are also responsible for stronger phonon dephasing for these transitions