We study many-body quantum dynamics using Floquet quantum circuits in one space dimension as simple examples of systems with local interactions that support ergodic phases. Physical properties can be expressed in terms of multiple sums over Feynman histories, which for these models are paths or many-body orbits in Fock space. A natural simplification of such sums is the diagonal approximation, where the only terms that are retained are ones in which each path is paired with a partner that carries the complex conjugate weight. We identify the regime in which the diagonal approximation holds, and the nature of the leading corrections to it. We focus on the behaviour of the spectral form factor (SFF) and of matrix elements of local operators, averaged over an ensemble of random circuits, making comparisons with the predictions of random matrix theory (RMT) and the eigenstate thermalisation hypothesis (ETH). We show that properties are dominated at long times by contributions to orbit sums in which each orbit is paired locally with a conjugate, as in the diagonal approximation, but that in large systems these contributions consist of many spatial domains, with distinct local pairings in neighbouring domains. The existence of these domains is reflected in deviations of the SFF from RMT predictions, and of matrix element correlations from ETH predictions; deviations of both kinds diverge with system size. We demonstrate that our physical picture of orbit-pairing domains has a precise correspondence in the spectral properties of a transfer matrix that acts in the space direction to generate the ensemble-averaged SFF. In addition, we find that domains of a second type control non-Gaussian fluctuations of the SFF. These domains are separated by walls which are related to the entanglement membrane, known to characterise the scrambling of quantum information.