For a generic quantum many-body system, the quantum ergodic regime is defined as the limit in which the spectrum of the system resembles that of a random matrix theory (RMT) in the corresponding symmetry class. In this paper we analyse the time dependence of correlation functions of operators. We study them in the ergodic limit as well as their approach to the ergodic limit which is controlled by non-universal massive modes. An effective field theory (EFT) corresponding to the causal symmetry and its breaking describes the ergodic phase. We demonstrate that the resulting Goldstone-mode theory has a topological expansion, analogous to the one described in arXiv:2008.02271 with added operator sources, whose leading non-trivial topologies give rise to the universal ramp seen in correlation functions. The ergodic behaviour of operators in our EFT is seen to result from a combination of RMT-like spectral statistics and Haar averaging over wave-functions. Furthermore we analytically capture the plateau behaviour by taking into account the contribution of a second saddle point. Our main interest are quantum many-body systems with holographic duals and we explicitly establish the validity of the EFT description in the SYK-class of models, starting from their microscopic description. By studying the tower of massive modes above the Goldstone sector we get a detailed understanding of how the ergodic EFT phase is approached and derive the relevant Thouless time scales. We point out that the topological expansion can be reinterpreted in terms of contributions of bulk wormholes and baby-universes.