We present a rest-frame ultraviolet morphological analysis of 108 z=2.1 Lyman Alpha Emitters (LAEs) in the Extended Chandra Deep Field South (ECDF-S) and compare it to a similar sample of 171 LAEs at z=3.1. Using Hubble Space Telescope (HST) images from the Galaxy Evolution from Morphology and SEDs survey, Great Observatories Origins Deep Survey, and Hubble Ultradeep Field, we measure size and photometric component distributions, where photometric components are defined as distinct clumps of UV-continuum emission. At both redshifts, the majority of LAEs have observed half-light radii <~ 2 kpc, but the median half-light radius rises from 1.0 kpc at z=3.1 to 1.4 kpc at z=2.1. A similar evolution is seen in the sizes of individual rest-UV components, but there is no evidence for evolution in the number of multi-component systems. In the z=2.1 sample, we see clear correlations between the size of an LAE and other physical properties derived from its SED. LAEs are found to be larger for galaxies with higher stellar mass, star formation rate, and dust obscuration, but there is no evidence for a trend between equivalent width and half-light radius at either redshift. The presence of these correlations suggests that a wide range of objects are being selected by LAE surveys at z~2, including a significant fraction of objects for which a massive and moderately extended population of old stars underlies the young starburst giving rise to the Lyman alpha emission.