We quantify the fraction of the cosmic infrared background (CIB) that originates from galaxies identified in the UV/optical/near-infrared by stacking 81,250 (~35.7 arcmin^2) K-selected sources (K_AB < 24.0), split according to their rest-frame U - V vs. V - J colors into 72,216 star-forming and 9,034 quiescent galaxies, on maps from Spitzer/MIPS (24um), Herschel/SPIRE (250, 350, 500um), Herschel/PACS (100, 160um), and AzTEC (1100um). The fraction of the CIB resolved by our catalog is (69 $pm$ 15)% at 24um, (78 $pm$ 17)% at 70um, (58 $pm$ 13)% at 100um, (78 $pm$ 18)% at 160um, (80 $pm$ 17)% at 250um, (69 $pm$ 14)% at 350um, (65 $pm$ 12)% at 500um, and (45 $pm$ 8)% at 1100um. Of that total, about 95% originates from star-forming galaxies, while the remaining 5% is from apparently quiescent galaxies. The CIB at $lambda$ < 200um is sourced predominantly from galaxies at z < 1, while at $lambda$ > 200um the bulk originates from 1 < z < 2. Galaxies with stellar masses log(M/ M_sun)=9.5-11 are responsible for the majority of the CIB, with those in the log(M/ M_sun)=9.5-10 contributing mostly at $lambda$ < 250um, and those in the log(M/ M_sun)=10.5-11 bin dominating at $lambda$ > 350um. The contribution from galaxies in the log(M/ M_sun)=9.0-9.5 and log(M/ M_sun)=11.0-12.0 stellar mass bins contribute the least, both of order 5%, although the highest stellar-mass bin is a significant contributor to the luminosity density at z > 2. The luminosities of the galaxies responsible for the CIB shifts from a combination of normal and luminous infrared galaxies (LIRGs) at $lambda$ < 160um, to LIRGs at 160um < $lambda$ < 500um, to finally LIRGs and ultra-luminous infrared galaxies (ULIRGs) at $lambda$ > 500um. Stacking analyses were performed with SIMSTACK (available at http://www.astro.caltech.edu/~viero/viero_homepage/toolbox.html) which accounts for possible biases due to clustering.