We combine the recent determination of the evolution of the cosmic density of molecular gas (H_2) using deep, volumetric surveys, with previous estimates of the cosmic density of stellar mass, star formation rate and atomic gas (HI), to constrain the evolution of baryons associated with galaxies averaged over cosmic time and space. The cosmic HI and H_2 densities are roughly equal at z~1.5. The H_2 density then decreases by a factor 6^{+3}_{-2} to todays value, whereas the HI density stays approximately constant. The stellar mass density is increasing continuously with time and surpasses that of the total gas density (HI and H_2) at redshift z~1.5. The growth in stellar mass cannot be accounted for by the decrease in cosmic H_2 density, necessitating significant accretion of additional gas onto galaxies. With the new H_2 constraints, we postulate and put observational constraints on a two step gas accretion process: (i) a net infall of ionized gas from the intergalactic/circumgalactic medium to refuel the extended HI reservoirs, and (ii) a net inflow of HI and subsequent conversion to H_2 in the galaxy centers. Both the infall and inflow rate densities have decreased by almost an order of magnitude since z~2. Assuming that the current trends continue, the cosmic molecular gas density will further decrease by about a factor of two over the next 5 Gyr, the stellar mass will increase by approximately 10%, and cosmic star formation activity will decline steadily toward zero, as the gas infall and accretion shut down.