Seed black holes formed in the collapse of population III stars have been invoked to explain the presence of supermassive black holes at high redshift. It has been suggested that a seed black hole can grow up to $10^{5sim 6}sunm$ through highly super-Eddington accretion for a period of $sim 10^{6sim 7}$ yr between redshift $z=20sim 24$. We studied the feedback of radiation pressure, Compton heating and outflow during the seed black hole growth. It is found that its surrounding medium fueled to the seed hole is greatly heated by Compton heating. For a super-critical accretion onto a $10^3sunm$ seed hole, a Compton sphere (with a temperature $sim 10^6$K) forms in a timescale of $1.6times 10^3$yr so that the hole is only supplied by a rate of $10^{-3}$ Eddington limit from the Compton sphere. Beyond the Compton sphere, the kinetic feedback of the strong outflow heats the medium at large distance, this leads to a dramatical decrease of the outer Bondi accretion onto the black hole and avoid the accumulation of the matter. The highly super-critical accretion will be rapidly halted by the strong feedback. The seed black holes hardly grow up at the very early universe unless the strong feedback can be avoided.