We present results from an on-going follow-up campaign of far-infrared sources detected as part of our ISOPHOT Cosmic IR Background project. Fields have been imaged in the optical and near-infrared, and we find at least a third of the FIR targets areas to contain a bright and nearby star-forming galaxy. We also explore the largely neglected possibility that instead of individual galaxies some of the fainter FIR sources are confused sums of several sources - or even whole cores of galaxy clusters at redshifts of z sim 0.4-0.8. We look for correlations in the FIR positions with extremely red objects (EROs) and significant peaks in the galaxy surface density and peaks in cluster red sequence signal. Several matches are found and we have set out to study cluster candidates spectroscopically. The campaign is producing an interesting base to study IR-luminous, strongly star-forming galaxies in potential cluster environments.