The co-evolution of supermassive black holes (SMBHs) with their host galaxies remains to be fully explored, especially at high redshift. While often understood as a consequence of self-regulation via AGN feedback, it may also be explained by alternative SMBH accretion models. Here, we expand on previous work by studying the growth of SMBHs with the help of a large suite of cosmological zoom-in simulations (textsc{small{MassiveFIRE}}) that are part of the Feedback in Realistic Environments (FIRE) project. The growth of SMBHs is modeled in post-processing with different accretion models, placements, and merger treatments, and validated by comparing to on-the-fly calculations. Scaling relations predicted by the gravitational torque driven accretion (GTDA) model agree with observations at low redshift emph{without} the need for AGN feedback, in contrast to models in which the accretion rate depends strongly on SMBH mass. At high redshift, we find deviations from the local scaling relations in line with previous results. In particular, SMBHs are under-massive, presumably due to stellar feedback, but start to grow efficiently once their host galaxies reach $M_* sim 10^{10} M_{odot}$. We analyze and explain these findings in the context of a simple analytic model. Finally, we show that the predicted scaling relations depend sensitively on the efficiency of SMBH merging. These findings highlight the relevance of understanding the evolution of SMBH-galaxy scaling relations to predict the rate of gravitational wave signals from SMBH mergers across cosmic history.