We study the evolution of the number density, as a function of the size, of passive early-type galaxies with a wide range of stellar masses 10^10<M*/Msun<10^11.5) from z~3 to z~1, exploiting the unique dataset available in the GOODS-South field, including the recently obtained WFC3 images as a part of the Cosmic Assembly Near-infrared Deep Extragalactic Legacy Survey (CANDELS). In particular, we select a sample of 107 massive (M*>10^10 M_sun), passive (SSFR<10^-2 Gyr^-1) and morphologically spheroidal galaxies at 1.2<z<3, taking advantage of the panchromatic dataset available for GOODS, including VLT, CFHT, Spitzer, Chandra and HST ACS+WFC3 data. We find that at 1<z<3 the passively evolving early-type galaxies are the reddest and most massive objects in the Universe, and we prove that a correlation between mass, morphology, color and star-formation activity is already in place at that epoch. We measure a significant evolution in the mass-size relation of passive early-type galaxies (ETGs) from z~3 to z~1, with galaxies growing on average by a factor of 2 in size in a 3 Gyr timescale only. We witness also an increase in the number density of passive ETGs of 50 times over the same time interval. We find that the first ETGs to form at z>2 are all compact or ultra-compact, while normal sized ETGs (meaning ETGs with sizes comparable to those of local counterparts of the same mass) are the most common ETGs only at z<1. The increase of the average size of ETGs at 0<z<1 is primarily driven by the appearance of new large ETGs rather than by the size increase of individual galaxies.