Using deep NIR VLT/ISAAC and optical HST/WFPC2 imaging in the fields of the HDFS and MS1054-03, we study the rest-frame UV-to-optical colors and magnitudes of galaxies to z~3. While there is no evidence for a red sequence at z~3, there does appear to
be a well-defined color-magnitude relation (CMR) for blue galaxies at all redshifts, with more luminous galaxies having redder U-V colors. The slope of the blue CMR is independent of redshift d(U-V)/dMV = -0.09 (0.01) and can be explained by a correlation of dust-reddening with luminosity. The average color at fixed luminosity reddens strongly Delta(U-V) = 0.75 from z~3 to z=0, much of which can be attributed to aging of the stars. The color scatter of the blue sequence is relatively small sigma(U-V) = 0.25 (0.03) and constant to z~3, but notably asymmetrical with a sharp blue ridge and a wing towards redder colors. We explore sets of star formation histories to study the constraints placed by the shape of the scatter at z=2-3. One particular set of models, episodic star formation, reproduces the detailed properties very well. For a two-state model with high and low star formation, the duty cycle is constrained to be > 40% and the contrast between the states must be a factor > 5 (or a scatter in log(SFR) of > 0.35 dex around the mean). However, episodic models do not explain the observed tail of very red galaxies, primarily Distant Red Galaxies (DRGs), which may have ceased star formation altogether or are more heavily obscured. Finally, the relative number density of red, luminous MV < -20.5 galaxies increases by a factor of ~ 6 from z = 2.7 to z = 0.5, as does their contribution to the total rest-frame V-band luminosity density. We are likely viewing the progressive formation of red, passively evolving galaxies.
