Coherent generation of indistinguishable single photons is crucial for many quantum communication and processing protocols. Solid-state realizations of two-level atomic transitions or three-level spin-$Lambda$ systems offer significant advantages over their atomic counterparts for this purpose, albeit decoherence can arise due to environmental couplings. One popular approach to mitigate dephasing is to operate in the weak excitation limit, where excited state population is minimal and coherently scattered photons dominate over incoherent emission. Here we probe the coherence of photons produced using two-level and spin-$Lambda$ solid-state systems. We observe that the coupling of the atomic-like transitions to the vibronic transitions of the crystal lattice is independent of driving strength and detuning. We apply a polaron master equation to capture the non-Markovian dynamics of the ground state vibrational manifolds. These results provide insight into the fundamental limitations for photon coherence from solid-state quantum emitters, with the consequence that deterministic single-shot quantum protocols are impossible and inherently probabilistic approaches must be embraced.