We present the first calculation of the hadronic tensor on the lattice for the nucleon. The hadronic tensor can be used to extract the structure functions in deep inelastic scatterings and also provide information for the neutrino-nucleon scattering which is crucial to the neutrino-nucleus scattering experiments at low energies. The most challenging part in the calculation is to solve an inverse problem. We have implemented and tested three algorithms using mock data, showing that the Bayesian Reconstruction method has the best resolution in extracting peak structures while the Backus-Gilbert and Maximum Entropy methods are somewhat more stable for the flat spectral function. Numerical results are presented for both the elastic case (clover fermions on domain wall configuration with $m_pisim$ 370 MeV and $asim$ 0.06 fm) and a case (anisotropic clover lattice with $m_pisim$ 380 MeV and $a_tsim$ 0.035 fm) with large momentum transfer. For the former case, the reconstructed Minkowski hadronic tensor gives precisely the vector charge which proves the feasibility of the approach. While for the latter case, the nucleon resonances and possibly shallow inelastic scattering contributions around $ u=1$ GeV are clearly observed but no information is obtained for higher excited states with $ u>2$ GeV. A check of the effective masses of $rho$ meson with different lattice setups indicates that, in order to reach higher energy transfers, using lattices with smaller lattice spacings is essential.