We present a simple physical model for populating dark matter halos with Lyman Alpha Emiiters(LAEs) and predict the physical properties of LAEs at z~3-7. The central tenet of this model is that the Ly-alpha luminosity is proportional to the star formation rate (SFR) which is directly related to the halo mass accretion rate. The only free parameter in our model is then the star-formation efficiency (SFE). An efficiency of 2.5% provides the best-fit to the Ly-alpha luminosity function (LF) at redshift z=3.1, and we use this SFE to construct Ly-alpha LFs at other redshifts. Our model reproduce the Ly-alpha LFs, stellar ages, SFR ~1-10; Msun/yr, stellar masses ~ 10^7-10^8 Msun and the clustering properties of LAEs at z~3-7. We find the spatial correlation lengths ro ~ 3-6 Mpc/h, in agreement with the observations. Finally, we estimate the field-to-field variation ~ 30% for current volume and flux limited surveys, again consistent with observations. Our results suggest that the star formation, and hence Ly-alpha emission in LAEs is powered by the accretion of new material, and that the physical properties of LAEs do not evolve significantly over a wide range of redshifts. Relating the accreted mass, rather than the total mass of halos, to the Ly-alpha luminosity of LAEs naturally gives rise to the duty cycle of LAEs.