We investigate the stellar populations of Lyman alpha emitters (LAEs) at z=3.1 and 3.7 in 0.65 deg^2 of the Subaru/XMM-Newton Deep Field, based on rest-frame UV-to-optical photometry obtained from the Subaru/XMM-Newton Deep Survey, the UKIDSS/Ultra Deep Survey, and the Spitzer legacy survey of the UKIDSS/UDS. Among a total of 302 LAEs (224 for z=3.1 and 78 for z=3.7), only 11 are detected in the K band, i.e., brighter than K(3sigma)=24.1 mag. Eight of the 11 K-detected LAEs are spectroscopically confirmed. We find that the K-undetected objects, which should closely represent the LAE population as a whole, have low stellar masses of ~ 10^8 - 10^8.5 Msun, modest SFRs of 1 - 100 Msun yr^-1, and modest dust extinction of E(B-V) < 0.2. The K-detected objects are massive, Mstar ~ 10^9 - 10^10.5 Msun, and have significant dust extinction with a median of E(B-V) ~= 0.3. Four K-detected objects with the reddest spectral energy distributions, two of which are spectroscopically confirmed, are heavily obscured with E(B-V) ~ 0.65, and their continua resemble those of some local ULIRGs. Interestingly, they have large Lyman alpha equivalent widths ~= 70 - 250 A. If these four are excluded, our sample has a weak anti-correlation between EW(Lya) and Mstar. We compare the stellar masses and the specific star formation rates (sSFR) of LAEs with those of Lyman-break galaxies, distant red galaxies, submillimetre galaxies, and I- or K-selected galaxies with z_phot ~ 3. We find that the LAE population is the least massive among all the galaxy populations in question, but with relatively high sSFRs, while NIR-detected LAEs have Mstar and sSFR similar to LBGs. Our reddest four LAEs have very high sSFRs in spite of large Mstar, thus occupying a unique region in the Mstar versus sSFR space. (abridged)