We measure the detuning-dependent dynamics of a quasi-resonantly excited single quantum dot coupled to a micropillar cavity. The system is modeled with the dissipative Jaynes-Cummings model where all experimental parameters are determined by explicit measurements. We observe non-Markovian dynamics when the quantum dot is tuned into resonance with the cavity leading to a non-exponential decay in time. Excellent agreement between experiment and theory is observed with no free parameters providing the first quantitative description of an all-solid-state cavity QED system based on quantum dot emitters.