We study the link between observed ultraviolet luminosity, stellar mass, and dust attenuation within rest-frame UV-selected samples at z~ 4, 3, and 1.5. We measure by stacking at 250, 350, and 500 um in the Herschel/SPIRE images from the HerMES program the average infrared luminosity as a function of stellar mass and UV luminosity. We find that dust attenuation is mostly correlated with stellar mass. There is also a secondary dependence with UV luminosity: at a given UV luminosity, dust attenuation increases with stellar mass, while at a given stellar mass it decreases with UV luminosity. We provide new empirical recipes to correct for dust attenuation given the observed UV luminosity and the stellar mass. Our results also enable us to put new constraints on the average relation between star formation rate and stellar mass at z~ 4, 3, and 1.5. The star formation rate-stellar mass relations are well described by power laws (SFR~ M^0.7), with the amplitudes being similar at z~4 and z~3, and decreasing by a factor of 4 at z~1.5 at a given stellar mass. We further investigate the evolution with redshift of the specific star formation rate. Our results are in the upper range of previous measurements, in particular at z~3, and are consistent with a plateau at 3<z<4. Current model predictions (either analytic, semi-analytic or hydrodynamic) are inconsistent with these values, as they yield lower predictions than the observations in the redshift range we explore. We use these results to discuss the star formation histories of galaxies in the framework of the Main Sequence of star-forming galaxies. Our results suggest that galaxies at high redshift (2.5<z<4) stay around 1 Gyr on the Main Sequence. With decreasing redshift, this time increases such that z=1 Main Sequence galaxies with 10^8<M_*/Msun<10^10 stay on the Main Sequence until z=0.