We propose a characterization of quantum many-body chaos: given a collection of simple operators, the set of all possible pair-correlations between these operators can be organized into a matrix with random-matrix-like spectrum. This approach is particularly useful for locally interacting systems, which do not generically show exponential Lyapunov growth of out-of-time-ordered correlators. We demonstrate the validity of this characterization by numerically studying the Sachdev-Ye-Kitaev model and a one-dimensional spin chain with random magnetic field (XXZ model).