Understanding how seed black holes grow into intermediate and supermassive black holes (IMBHs and SMBHs, respectively) has important implications for the duty-cycle of active galactic nuclei (AGN), galaxy evolution, and gravitational wave astronomy. Most studies of the cosmological growth and merger history of black holes have used semianalytic models and have concentrated on SMBH growth in luminous galaxies. Using high resolution cosmological N-body simulations, we track the assembly of black holes over a large range of final masses -- from seed black holes to SMBHs -- over widely varying dynamical histories. We used the dynamics of dark matter halos to track the evolution of seed black holes in three different gas accretion scenarios. We have found that growth of Sagittarius A* - size SMBH reaches its maximum mass M_{SMBH}~10^6Msun at z~6 through early gaseous accretion episodes, after which it stays at near constant mass. At the same redshift, the duty-cycle of the host AGN ends, hence redshift z=6 marks the transition from an AGN to a starburst galaxy which eventually becomes the Milky Way. By tracking black hole growth as a function of time and mass, we estimate that the IMBH merger rate reaches a maximum of R_{max}=55 yr^-1 at z=11. From IMBH merger rates we calculate N_{ULX}=7 per Milky Way type galaxy per redshift in redshift range 2<z<6.