The Hubble Ultra Deep Field (HUDF) contains a significant number of B, V and i-band dropout objects, many of which were recently confirmed to be young star-forming galaxies at z~4-6. These galaxies are too faint individually to accurately measure their radial surface brightness profiles. Their average light profiles are potentially of great interest, since they may contain clues to the time since the onset of significant galaxy assembly. We separately co-add V, i and z-band HUDF images of sets of z~4,5 and 6 objects, pre-selected to have nearly identical compact sizes and the roundest shapes. From these stacked images, we are able to study the averaged radial structure of these objects at much higher signal-to-noise ratio than possible for an individual faint object. Here we explore the reliability and usefulness of a stacking technique of compact objects at z~4-6 in the HUDF. Our results are: (1) image stacking provides reliable and reproducible average surface brightness profiles; (2) the shape of the average surface brightness profiles show that even the faintest z~4-6 objects are resolved; and (3) if late-type galaxies dominate the population of galaxies at z~4-6, as previous HST studies have shown, then limits to dynamical age estimates for these galaxies from their profile shapes are comparable with the SED ages obtained from the broadband colors. We also present accurate measurements of the sky-background in the HUDF and its associated 1-sigma uncertainties.