We investigate early black hole (BH) growth through the methodical search for $zgtrsim5$ AGN in the $Chandra$ Deep Field South. We base our search on the $Chandra$ 4-Ms data with flux limits of $9.1times 10^{-18}$ (soft, 0.5 - 2 keV) and $5.5times 10^{-17} mathrm{erg} mathrm{s}^{-1} mathrm{cm}^{-2}$ (hard, 2 - 8 keV). At $zsim5$ this corresponds to luminosities as low as $sim10^{42}$ ($sim10^{43}$) $mathrm{erg} mathrm{s}^{-1}$ in the soft (hard) band and should allow us to detect Compton-thin AGN with $M_mathrm{BH}>10^7 M_{odot}$ and Eddington ratios > 0.1. Our field ($0.03~mathrm{deg}^2$) contains over 600 $zsim5$ Lyman Break Galaxies. Based on lower redshift relations we would expect $sim20$ of them to host AGN. After combining the $Chandra$ data with GOODS/ACS, CANDELS/WFC3 and $Spitzer$/IRAC data, the sample consists of 58 high-redshift candidates. We run a photometric redshift code, stack the GOODS/ACS data, apply colour criteria and the Lyman Break Technique and use the X-ray Hardness Ratio. We combine our tests and using additional data find that all sources are most likely at low redshift. We also find five X-ray sources without a counterpart in the optical or infrared which might be spurious detections. We conclude that our field does not contain any convincing $zgtrsim5$ AGN. Explanations for this result include a low BH occupation fraction, a low AGN fraction, short, super-Eddington growth modes, BH growth through BH-BH mergers or in optically faint galaxies. By searching for $zgtrsim5$ AGN we are setting the foundation for constraining early BH growth and seed formation scenarios.