We constrain the total accreted mass density in supermassive black holes at z>6, inferred via the upper limit derived from the integrated X-ray emission from a sample of photometrically selected galaxy candidates. Studying galaxies obtained from the deepest Hubble Space Telescope images combined with the Chandra 4 Msec observations of the Chandra Deep Field South, we achieve the most restrictive constraints on total black hole growth in the early Universe. We estimate an accreted mass density <1000Mo Mpc^-3 at z~6, significantly lower than the previous predictions from some existing models of early black hole growth and earlier prior observations. These results place interesting constraints on early black growth and mass assembly by accretion and imply one or more of the following: (1) only a fraction of the luminous galaxies at this epoch contain active black holes; (2) most black hole growth at early epochs happens in dusty and/or less massive - as yet undetected - host galaxies; (3) there is a significant fraction of low-z interlopers in the galaxy sample; (4) early black hole growth is radiatively inefficient, heavily obscured and/or is due to black hole mergers as opposed to accretion or (5) the bulk of the black hole growth occurs at late times. All of these possibilities have important implications for our understanding of high redshift seed formation models.