We use morphological information of X-ray selected AGN hosts to set limits on the fraction of the accretion density of the Universe at z~1 that is not likely to be associated with major mergers. Deep X-ray observations are combined with high resolution optical data from the Hubble Space Telescope in the AEGIS, GOODS North and GOODS South fields to explore the morphological breakdown of X-ray sources in the redshift interval 0.5<z<1.3. The sample is split into disks, early-type bulge dominated galaxies, peculiar systems and point-sources in which the nuclear source outshines the host galaxy. The X-ray luminosity function and luminosity density of AGN at z~1 are then calculated as a function of morphological type. We find that disk-dominated hosts contribute 30pm9 per cent to the total AGN space density and 23pm6 per cent to the luminosity density at z~1. We argue that AGN in disk galaxies are most likely fueled not by major merger events but by minor interactions or internal instabilities. We find evidence that these mechanisms may be more efficient in producing luminous AGN (L_X>1e44 erg/s) compared to predictions for the stochastic fueling of massive black holes in disk galaxies.