We investigate the stellar populations of a sample of 162 Lyman-alpha emitting galaxies (LAEs) at z = 3.1 in the Extended Chandra Deep Field South, using deep Spitzer IRAC data available from the GOODS and SIMPLE surveys to derive reliable stellar population estimates. We divide the LAEs according to their rest-frame near-IR luminosities into IRAC-detected and IRAC-undetected samples. About 70% of the LAEs are undetected in 3.6 micron down to [3.6] = 25.2 AB. Stacking analysis reveals that the average stellar population of the IRAC-undetected sample has an age of ~ 200 Myr and a mass of ~ 3x10^8 solar masses, consistent with the expectation that LAEs are mostly young and low-mass galaxies. On the other hand, the IRAC-detected LAEs are on average significantly older and more massive, with an average age > 1 Gyr and mass ~ 10^10 solar masses. Comparing the IRAC colors and magnitudes of the LAEs to z ~ 3 Lyman break galaxies (LBGs) shows that the IRAC-detected LAEs lie at the faint blue end of the LBG color-magnitude distribution, suggesting that IRAC-detected LAEs may be the low mass extension of the LBG population. We also present tentative evidence for a small fraction (~ 5%) of obscured AGN within the LAE sample. Our results suggest that LAEs posses a wide range of ages and masses. Additionally, the presence of evolved stellar populations inside LAEs suggests that the Lyman-alpha luminous phase of galaxies may either be a long-lasting or recurring phenomenon.