Chandra data in the COSMOS, AEGIS-XD and 4Ms CDFS are combined with optical/near-IR photometry to determine the rest-frame U-V vs V-J colours of X-ray AGN hosts at mean redshifts 0.40 and 0.85. This combination of colours (UVJ) provides an efficient means of separating quiescent from star-forming, including dust reddened, galaxies. Morphological information emphasises differences between AGN split by their UVJ colours. AGN in quiescent galaxies are dominated by spheroids, while star-forming hosts are split between bulges and disks. The UVJ diagram of AGN hosts is then used to set limits on the accretion density associated with evolved and star-forming systems. Most of the black hole growth since z~1 is associated with star-forming hosts. Nevertheless, ~15-20% of the X-ray luminosity density since z~1, is taking place in the quiescent region of the UVJ diagram. For the z~0.40 subsample, there is tentative evidence (2sigma significance), that AGN split by their UVJ colours differ in Eddington ratio. AGN in star-forming hosts dominate at high Eddington ratios, while AGN in quiescent hosts become increasingly important as a fraction of the total population toward low Eddington ratios. At higher redshift, z~0.8, such differences are significant at the 2sigma level only at Eddington ratios >1e-3. These findings are consistent with scenarios in which diverse accretion modes are responsible for the build-up of SMBHs at the centres of galaxies. We compare our results with the GALFORM semi-analytic model, which postulates two black hole fuelling modes, the first linked to star-formation and the second occuring in passive galaxies. GALFORM predicts a larger fraction of black hole growth in quiescent galaxies at z<1, compared to the data. Relaxing the strong assumption of the model that passive AGN hosts have zero star-formation rate could reconcile this disagreement.