We present nucleon elastic scattering calculation based on Greens function formalism in the Random-Phase Approximation. For the first time, the Gogny effective interaction is used consistently throughout the whole calculation to account for the complex, non-local and energy-dependent optical potential. Effects of intermediate single-particle resonances are included and found to play a crucial role in the account for measured reaction cross section. Double counting of the particle-hole second-order contribution is carefully addressed. The resulting integro-differential Schrodinger equation for the scattering process is solved without localization procedures. The method is applied to neutron and proton elastic scattering from $^{40}$Ca. A successful account for differential and integral cross sections, including analyzing powers, is obtained for incident energies up to 30 MeV. Discrepancies at higher energies are related to much too high volume integral of the real potential for large partial waves. Moreover, this works opens the way for future effective interactions suitable simultaneously for both nuclear structure and reaction.