A key unresolved question is the role that galaxy mergers play in driving stellar mass growth over cosmic time. Recent observational work hints at the possibility that the overall contribution of `major mergers (mass ratios $gtrsim$1:4) to cosmic stellar mass growth may be small, because they enhance star formation rates by relatively small amounts at high redshift, when much of todays stellar mass was assembled. However, the heterogeneity and relatively small size of todays datasets, coupled with the difficulty in identifying genuine mergers, makes it challenging to $textit{empirically}$ quantify the merger contribution to stellar mass growth. Here, we use Horizon-AGN, a cosmological hydrodynamical simulation, to comprehensively quantify the contribution of mergers to the star formation budget over the lifetime of the Universe. We show that: (1) both major and minor mergers enhance star formation to similar amounts, (2) the fraction of star formation directly attributable to merging is small at all redshifts (e.g. $sim$35 and $sim$20 per cent at z$sim$3 and z$sim$1 respectively) and (3) only $sim$25 per cent of todays stellar mass is directly attributable to galaxy mergers over cosmic time. Our results suggest that smooth accretion, not merging, is the dominant driver of stellar mass growth over the lifetime of the Universe.