The extragalactic background light (EBL) is of fundamental importance both for understanding the entire process of galaxy evolution and for gamma-ray astronomy, but the overall spectrum of the EBL between 0.1-1000 microns has never been determined directly from galaxy spectral energy distribution (SED) observations over a wide redshift range. The evolving, overall spectrum of the EBL is derived here utilizing a novel method based on observations only. This is achieved from the observed evolution of the rest-frame K-band galaxy luminosity function up to redshift 4 (Cirasuolo et al. 2010), combined with a determination of galaxy SED-type fractions. These are based on fitting SWIRE templates to a multiwavelength sample of about 6000 galaxies in the redshift range from 0.2 to 1 from the All-wavelength Extended Groth Strip International Survey (AEGIS). The changing fractions of quiescent galaxies, star-forming galaxies, starburst galaxies and AGN galaxies in that redshift range are estimated, and two alternative extrapolations of SED-types to higher redshifts are considered. This allows calculation of the evolution of the luminosity densities from the UV to the IR, the evolving star formation rate density of the universe, the evolving contribution to the bolometric EBL from the different galaxy populations including AGN galaxies and the buildup of the EBL. Our EBL calculations are compared with those from a semi-analytic model, from another observationally-based model and observational data. The EBL uncertainties in our modeling based directly on the data are quantified, and their consequences for attenuation of very high energy gamma-rays due to pair production on the EBL are discussed. It is concluded that the EBL is well constrained from the UV to the mid-IR, but independent efforts from infrared and gamma-ray astronomy are needed in order to reduce the uncertainties in the far-IR.