Differential dynamic microscopy (DDM) is a form of video image analysis that combines the sensitivity of scattering and the direct visualization benefits of microscopy. DDM is broadly useful in determining dynamical properties including the intermediate scattering function for many spatiotemporally correlated systems. Despite its straightforward analysis, DDM has not been fully adopted as a routine characterization tool, largely due to computational cost and lack of algorithmic robustness. We present a comprehensive statistical framework that aims at quantifying error, reducing the computational order and enhancing the robustness of DDM analysis. We quantify the error, and propagate an independent noise term to derive a closed-form expression of the expected value and variance of the observed image structure function. Significantly, we propose an unbiased estimator of the mean of the noise in the observed image structure function, which can be determined experimentally and significantly improves the accuracy of applications of DDM. Furthermore, through use of Gaussian Process Regression (GPR), we find that predictive samples of the image structure function require only around 1% of the Fourier Transforms of the observed quantities. This vastly reduces computational cost, while preserving information of the quantities of interest, such as quantiles of the image scattering function, for subsequent analysis. The approach, which we call DDM with Uncertainty Quantification (DDM-UQ), is validated using both simulations and experiments with respect to accuracy and computational efficiency, as compared with conventional DDM and multiple particle tracking. Overall, we propose that DDM-UQ lays the foundation for important new applications of DDM, as well as to high-throughput characterization.