By using cosmological hydrodynamical simulations we study the effect of supernova (SN) and active galactic nuclei (AGN) feedback on the mass transport of gas on to galactic nuclei and the black hole (BH) growth down to redshift z~6. We study the BH growth in relation with the mass transport processes associated with gravity and pressure torques, and how they are modified by feedback. Cosmological gas funelled through cold flows reaches the galactic outer region close to free-fall. Then torques associated to pressure triggered by gas turbulent motions produced in the circum-galactic medium by shocks and explosions from SNe are the main source of mass transport beyond the central ~ 100 pc. Due to high concentrations of mass in the central galactic region, gravitational torques tend to be more important at high redshift. The combined effect of almost free-falling material and both gravity and pressure torques produces a mass accretion rate of order ~ 1 M_sun/yr at ~ pc scales. In the absence of SN feedback, AGN feedback alone does not affect significantly either star formation or BH growth until the BH reaches a sufficiently high mass of $sim 10^6$ M_sun to self-regulate. SN feedback alone, instead, decreases both stellar and BH growth. Finally, SN and AGN feedback in tandem efficiently quench the BH growth, while star formation remains at the levels set by SN feedback alone due to the small final BH mass, ~ few 10^5 M_sun. SNe create a more rarefied and hot environment where energy injection from the central AGN can accelerate the gas further.