We present new information on galaxies in the vicinity of luminous radio galaxies and quasars at z=4,5,6. These fields were previously found to contain overdensities of Lyman Break Galaxies (LBGs) or spectroscopic Lyman alpha emitters. We use HST and Spitzer data to infer stellar masses, and contrast our results with large samples of LBGs in more average environments as probed by the Great Observatories Origins Deep Survey (GOODS). The following results were obtained. First, LBGs in both overdense regions and in the field at z=4-5 lie on a very similar sequence in a z-[3.6] versus [3.6] color-magnitude diagram. This is interpreted as a sequence in stellar mass (log[M*/Msun] = 9-11) in which galaxies become increasingly red due to dust and age as their star formation rate (SFR) increases. Second, the two radio galaxies are among the most massive objects (log[M*/Msun]~11) known to exist at z~4-5, and are extremely rare based on the low number density of such objects as estimated from the ~25x larger area GOODS survey. We suggest that the presence of these massive galaxies and supermassive black holes has been boosted through rapid accretion of gas or merging inside overdense regions. Third, the total stellar mass found in the z=4 ``proto-cluster TN1338 accounts for <30% of the stellar mass on the cluster red sequence expected to have formed at z>4, based on a comparison with the massive X-ray cluster Cl1252 at z=1.2. Although future near-infrared observations should determine whether any massive galaxies are currently being missed, one possible explanation for this mass difference is that TN1338 evolves into a smaller cluster than Cl1252. This raises the interesting question of whether the most massive protocluster regions at z>4 remain yet to be discovered.