Understanding the processes that drive the formation of black holes (BHs) is a key topic in observational cosmology. While the observed $M_{mathrm{BH}}$--$M_{mathrm{Bulge}}$ correlation in bulge-dominated galaxies is thought to be produced by major mergers, the existence of a $M_{mathrm{BH}}$--$M_{star}$ relation, across all galaxy morphological types, suggests that BHs may be largely built by secular processes. Recent evidence that bulge-less galaxies, which are unlikely to have had significant mergers, are offset from the $M_{mathrm{BH}}$--$M_{mathrm{Bulge}}$ relation, but lie on the $M_{mathrm{BH}}$--$M_{star}$ relation, has strengthened this hypothesis. Nevertheless, the small size and heterogeneity of current datasets, coupled with the difficulty in measuring precise BH masses, makes it challenging to address this issue using empirical studies alone. Here, we use Horizon-AGN, a cosmological hydrodynamical simulation to probe the role of mergers in BH growth over cosmic time. We show that (1) as suggested by observations, simulated bulge-less galaxies lie offset from the main $M_{mathrm{BH}}$--$M_{mathrm{Bulge}}$ relation, but on the $M_{mathrm{BH}}$--$M_{star}$ relation, (2) the positions of galaxies on the $M_{mathrm{BH}}$--$M_{star}$ relation are not affected by their merger histories and (3) only $sim$35 per cent of the BH mass in todays massive galaxies is directly attributable to merging -- the majority ($sim$65 per cent) of BH growth, therefore, takes place gradually, via secular processes, over cosmic time.