We provide a unified theoretical approach to the quantum dynamics of absorption of single photons and subsequent excitonic energy transfer in photosynthetic light-harvesting complexes. Our analysis combines a continuous mode <n>-photon quantum optical master equation for the chromophoric system with the hierarchy of equations of motion describing excitonic dynamics in presence of non-Markovian coupling to vibrations of the chromophores and surrounding protein. We apply the approach to simulation of absorption of single-photon coherent states by pigment-protein complexes containing between one and seven chromophores, and compare with results obtained by excitation using a thermal radiation field. We show that the values of excitation probability obtained under single-photon absorption conditions can be consistently related to bulk absorption cross-sections. Analysis of the timescale and efficiency of single-photon absorption by light-harvesting systems within this full quantum description of pigment-protein dynamics coupled to a quantum radiation field reveals a non-trivial dependence of the excitation probability and the excited state dynamics induced by exciton-phonon coupling during and subsequent to the pulse, on the bandwidth of the incident photon pulse. For bandwidths equal to the spectral bandwidth of Chlorophyll a, our results yield an estimation of an average time of ~0.09 s for a single chlorophyll chromophore to absorb the energy equivalent of one (single-polarization) photon under irradiation by single-photon states at the intensity of sunlight.