To study the stellar population of local infrared galaxies, which contain star-forming galaxies, composite galaxies, LINERs, and Seyfert 2s. We also want to find whether infrared luminosity and spectral class have any effects on their stellar populations. The sample galaxies are selected from the main galaxy sample of SDSS-DR4 and then cross-correlated with the IRAS-PSCz catalog. We fit our spectra (stellar absorption lines and continua) using the spectral synthesis code STARLIGHT on the base of the templates of Simple Stellar Population and the spectra of star clusters.Among the 4 spectral classes, LINERs present the oldest stellar populations, and the other 3 sub-samples all present substantial young and intermediate age populations and very few old populations. The importance of young populations decreases from star-forming, composite, Seyfert 2 to LINER. As to different infrared luminosity bins, ULIGs & LIGs (log($L_{IR}/L_{odot})geq$11) present younger populations than starbursts and normal galaxies. However, the dominant contributors to mass are old populations in all sample galaxies. The fittings by using the spectra of star clusters with different ages and metallicities as templates also give consistent results. The dominated populations in star-forming and composite galaxies are those with metallicity $Z=0.2Z_odot$, while LINERs and Seyfert 2s are more metal-rich. The normal galaxies are more metal-rich than the ULIGs & LIGs and starbursts for the star-forming galaxies within different infrared luminosity bins. Additionally, we also compare some synthesis results with other parameters obtained from the MPA/JHU catalog.