Ly$alpha$ photons scattered by neutral hydrogen atoms in the circumgalactic media or produced in the halos of star-forming galaxies are expected to lead to extended Ly$alpha$ emission around galaxies. Such low surface brightness Ly$alpha$ halos (LAHs) have been detected by stacking Ly$alpha$ images of high-redshift star-forming galaxies. We study the origin of LAHs by performing radiative transfer modeling of nine $z=3.1$ Lyman-Alpha Emitters (LAEs) in a high resolution hydrodynamic cosmological galaxy formation simulation. We develop a method of computing the mean Ly$alpha$ surface brightness profile of each LAE by effectively integrating over many different observing directions. Without adjusting any parameters, our model yields an average Ly$alpha$ surface brightness profile in remarkable agreement with observations. We find that observed LAHs cannot be accounted for solely by photons originating from the central LAE and scattered to large radii by hydrogen atoms in the circumgalactic gas. Instead, Ly$alpha$ emission from regions in the outer halo is primarily responsible for producing the extended LAHs seen in observations, which potentially includes both star-forming and cooling radiation. With the limit on the star formation contribution set by the ultra-violet (UV) halo measurement, we find that cooling radiation can play an important role in forming the extended LAHs. We discuss the implications and caveats of such a picture.