We study the far-infrared properties of 498 Lyman Alpha Emitters (LAEs) at z=2.8, 3.1 and 4.5 in the Extended Chandra Deep Field-South, using 250, 350 and 500 micron data from the Herschel Multi-tiered Extragalactic Survey (HerMES) and 870 micron data from the LABOCA ECDFS Submillimeter Survey (LESS). None of the 126, 280 or 92 LAEs at z=2.8, 3.1 and 4.5, respectively, are individually detected in the far-infrared data. We use stacking to probe the average emission to deeper flux limits, reaching $1sigma$ depths of ~0.1 to 0.4 mJy. The LAEs are also undetected at $ge3sigma$ in the stacks, although a $2.5sigma$ signal is observed at 870 micron for the z=2.8 sources. We consider a wide range of far-infrared spectral energy distributions (SEDs), including a M82 and an Sd galaxy template, to determine upper limits on the far-infrared luminosities and far-infrared-derived star-formation rates of the LAEs. These star-formation rates are then combined with those inferred from the Ly$alpha$ and UV emission to determine lower limits on the LAEs Ly$alpha$ escape fraction ($f_{rm esc}($Ly$alpha$)). For the Sd SED template, the inferred LAEs $f_{rm esc}($Ly$alpha$) are $gtrsim30%$ ($1sigma$) at z=2.8, 3.1 and 4.5, which are all significantly higher than the global $f_{rm esc}($Ly$alpha$) at these redshifts. Thus, if the LAEs $f_{rm esc}($Ly$alpha$) follows the global evolution then they have warmer far-infrared SEDs than the Sd galaxy template. The average and M82 SEDs produce lower limits on the LAE $f_{rm esc}($Ly$alpha$) of ~10 to 20% ($1sigma$), all of which are slightly higher than the global evolution of $f_{rm esc}($Ly$alpha$) but consistent with it at the 2 to 3$sigma$ level.