The long timescale evolution of a self-gravitating system is generically driven by two-body encounters. In many cases, the motion of the particles is primarily governed by the mean field potential. When this potential is integrable, particles move on nearly fixed orbits, which can be described in terms of angle-action variables. The mean field potential drives fast orbital motions (angles) whose associated orbits (actions) are adiabatically conserved on short dynamical timescales. The long-term stochastic evolution of the actions is driven by the potential fluctuations around the mean field and in particular by resonant two-body encounters, for which the angular frequencies of two particles are in resonance. We show that the stochastic gravitational fluctuations acting on the particles can generically be described by a correlated Gaussian noise. Using this approach, the so-called $eta$-formalism, we derive a diffusion equation for the actions in the test particle limit. We show that in the appropriate limits, this diffusion equation is equivalent to the inhomogeneous Balescu-Lenard and Landau equations. This approach provides a new view of the resonant diffusion processes associated with long-term orbital distortions. Finally, by investigating the example of the Hamiltonian Mean Field Model, we show how the present method generically allows for alternative calculations of the long-term diffusion coefficients in inhomogeneous systems.