We present an exact diagonalization study of the self-energy of the two-dimensional Hubbard model. To increase the range of available cluster sizes we use a corrected t-J model to compute approximate Greens functions for the Hubbard model. This allows to obtain spectra for clusters with 18 and 20 sites. The self-energy has several `bands of poles with strong dispersion and extended incoherent continua with k-dependent intensity. We fit the self-energy by a minimal model and use this to extrapolate the cluster results to the infinite lattice. The resulting Fermi surface shows a transition from hole pockets in the underdoped regime to a large Fermi surface in the overdoped regime. We demonstrate that hole pockets can be completely consistent with the Luttinger theorem. Introduction of next-nearest neighbor hopping changes the self-energy stronlgy and the spectral function with nonvanishing next-nearest-neighbor hopping in the underdoped region is in good agreement with angle resolved photoelectron spectroscopy.