Inelastic scattering experiments are key methods for mapping the full dispersion of fundamental excitations of solids in the ground as well as non-equilibrium states. A quantitative analysis of inelastic scattering in terms of phonon excitations requires identifying the role of multi-phonon processes. Here, we develop an efficient first-principles methodology for calculating the {it all-phonon} quantum mechanical structure factor of solids. We demonstrate our method by obtaining unprecedented agreement between measurements and calculations of the diffuse diffraction patterns of black phosphorus, showing that multi-phonon scattering plays a substantial role. The present approach constitutes a pivotal advancement in the interpretation of static and time-resolved electron, X-ray, and neutron inelastic scattering data.