We present a semiclassical calculation of the generalized form factor which characterizes the fluctuations of matrix elements of the quantum operators in the eigenbasis of the Hamiltonian of a chaotic system. Our approach is based on some recently developed techniques for the spectral form factor of systems with hyperbolic and ergodic underlying classical dynamics and f=2 degrees of freedom, that allow us to go beyond the diagonal approximation. First we extend these techniques to systems with f>2. Then we use these results to calculate the generalized form factor. We show that the dependence on the rescaled time in units of the Heisenberg time is universal for both the spectral and the generalized form factor. Furthermore, we derive a relation between the generalized form factor and the classical time-correlation function of the Weyl symbols of the quantum operators.