We present a formalism to infer the presence of merging by comparing the time derivative of the observed galaxy stellar mass function (MF) to the change of the MF expected from the star formation rate (SFR) in galaxies as a function of mass and time. We present the SFR in as a function of stellar mass and time spanning 9<logM<12 and 0<z<5. We show that at z>=3 the average SFR, is a power law of stellar mass (SFR~M^0.6). The average SFR in the most massive objects at this redshift is 100-500 Msun/yr. At z~3, the SFR starts to drop at the high mass end. As z decreases further, the SFR drops at progressively lower masses (downsizing), dropping most rapidly for high mass (logM>11) galaxies. The mass at which the SFR starts to deviate from the power-law form (break mass) progresses smoothly from logM~13 at z~5 to logM~10.9 at z~0.5. The break mass evolves with redshift as M(z)=2.7x10^10 (1+z)^2.1. We directly observe a relationship between SFH and mass. More massive galaxies have steeper and earlier onsets of SF, their SFR peaks earlier and higher, and the following decay has a shorter e-folding time. The SFR observed in high mass galaxies at z~4 is sufficient to explain their rapid increase in number density. Within large uncertainties, at most 0.8 major mergers per Gyr are consistent with the high-z data, yet enough to transform most high mass objects into ellipticals contemporaneously with their major star formation episode. In contrast, at z<1.5 and at logM>11, mergers contribute 0.1-0.2 Gyr^-1 to the relative increase in number density (~1 major merger per massive object at 1.5>z>0). At 10<logM<11, galaxies are being preferably destroyed in mergers at high z, while at later times the change in their numbers turns positive. This shows the top-down buildup of the red sequence suggested by other observations.