Differences in the confinement of electrons and holes in quantum dots are shown to profoundly impact the magnitude of scattering with acoustic phonons in materials where crystal deformation shifts the conduction and valence band in the same direction. Using an extensive model that includes the non-Markovian nature of the phonon reservoir, we show how the effect may be addressed by photoluminescence excitation spectroscopy of a single quantum dot. We also investigate the implications for cavity QED, i.e. a coupled quantum dot-cavity system, and demonstrate that the phonon scattering may be strongly quenched. The quenching is explained by a balancing between the deformation potential interaction strengths and the carrier confinement and depends on the quantum dot shape. Numerical examples suggest a route towards engineering the phonon scattering.