A quantum dot coupled to an optical cavity has recently proven to be an excellent source of on-demand single photons. Typically, applications require simultaneous high efficiency of the source and quantum indistinguishability of the extracted photons. While much progress has been made both in suppressing background sources of decoherence and utilizing cavity-quantum electrodynamics to overcome fundamental limitations set by the intrinsic exciton-phonon scattering inherent in the solid-state platform, the role of the excitation pulse has been often neglected. We investigate quantitatively the factors associated with pulsed excitation that can limit simultaneous efficiency and indistinguishability, including excitation of multiple excitons, multi-photons, and pump-induced dephasing, and find for realistic single photon sources that these effects cause degradation of the source figures-of-merit comparable to that of phonon scattering. We also develop rigorous open quantum system polaron master equation models of quantum dot dynamics under a time-dependent drive which incorporate non-Markovian effects of both photon and phonon reservoirs, and explicitly show how coupling to a high Q-factor cavity suppresses multi-photon emission in a way not predicted by commonly employed models. We then use our findings to summarize the criteria that can be used for single photon source optimization.