We incorporate a model for black hole growth during galaxy mergers into the semi-analytical galaxy formation model based on Lambda-CDM proposed by Baugh et al. (2005). Our black hole model has one free parameter, which we set by matching the observed zeropoint of the local correlation between black hole mass and bulge luminosity. We present predictions for the evolution with redshift of the relationships between black hole mass and bulge properties. Our simulations reproduce the evolution of the optical luminosity function of quasars. We study the demographics of the black hole population and address the issue of how black holes acquire their mass. We find that the direct accretion of cold gas during starbursts is an important growth mechanism for lower mass black holes and at high redshift. On the other hand, the re-assembly of pre-existing black hole mass into larger units via merging dominates the growth of more massive black holes at low redshift. This prediction could be tested by future gravitational wave experiments. As redshift decreases, progressively less massive black holes have the highest fractional growth rates, in line with recent claims of downsizing in quasar activity.