The growth of galaxies is a key problem in understanding the structure and evolution of the universe. Galaxies grow their stellar mass by a combination of star formation and mergers, with a relative importance that is redshift dependent. Theoretical models predict quantitatively different contributions from the two channels; measuring these from the data is a crucial constraint. Exploiting the UltraVISTA catalog and a unique sample of progenitors of local ultra massive galaxies selected with an abundance matching approach, we quantify the role of the two mechanisms from z=2 to 0. We also compare our results to two independent incarnations of semi-analytic models. At all redshifts, progenitors are found in a variety of environments, ranging from being isolated to having 5-10 companions with mass ratio at least 1:10 within a projected radius of 500 kpc. In models, progenitors have a systematically larger number of companions, entailing a larger mass growth for mergers than in observations, at all redshifts. Generally, in both observations and models, the inferred and the expected mass growth roughly agree, within the uncertainties. Overall, our analysis confirms the model predictions, showing how the growth history of massive galaxies is dominated by in situ star formation at z~2, both star-formation and mergers at 1<z<2, and by mergers alone at z<1. Nonetheless, detailed comparisons still point out to tensions between the expected mass growth and our results, which might be due to either an incorrect progenitors-descendants selection, uncertainties on star formation rate and mass estimates, or the adopted assumptions on merger rates.