We present a comparison between the observed color distribution, number and mass density of massive galaxies at 1.5 < z < 3 and a model by Hopkins et al. that relates the quasar and galaxy population on the basis of gas-rich mergers. In order to test the hypothesis that quiescent red galaxies are formed after a gas-rich merger involving quasar activity, we confront photometry of massive (M > 4x10^10 Msun) galaxies extracted from the FIRES, GOODS-South, and MUSYC surveys, together spanning an area of 496 arcmin^2, with synthetic photometry from hydrodynamical merger simulations. As in the Hopkins et al. (2006b) model, we use the observed quasar luminosity function to estimate the merger rate. We find that the synthetic U-V and V-J colors of galaxies that had a quasar phase in their past match the colors of observed galaxies that are best characterized by a quiescent stellar population. At z ~ 2.6, the observed number and mass density of quiescent red galaxies with M > 4x10^10 Msun is consistent with the model in which every quiescent massive galaxy underwent a quasar phase in the past. At z ~ 1.9, 2.8 times less quiescent galaxies are observed than predicted by the model as descendants of higher redshift quasars. The merger model also predicts a large number of galaxies undergoing merger-driven star formation. We find that the predicted number and mass density accounts for 30-50% of the observed massive star-forming galaxies. However, their colors do not match those of observed star-forming galaxies. In particular, the colors of dusty red galaxies are not reproduced by the simulations. Several possible origins of this discrepancy are discussed. The observational constraints on the validity of the model are currently limited by cosmic variance and uncertainties in stellar population synthesis and radiative transfer.