We study the structural evolution of massive galaxies by linking progenitors and descendants at a constant cumulative number density of n_c=1.4x10^{-4} Mpc^{-3} to z~3. Structural parameters were measured by fitting Sersic profiles to high resolution CANDELS HST WFC3 J_{125} and H_{160} imaging in the UKIDSS-UDS at 1<z<3 and ACS I_{814} imaging in COSMOS at 0.25<z<1. At a given redshift, we selected the HST band that most closely samples a common rest-frame wavelength so as to minimize systematics from color gradients in galaxies. At fixed n_c, galaxies grow in stellar mass by a factor of ~3 from z~3 to z~0. The size evolution is complex: galaxies appear roughly constant in size from z~3 to z~2 and then grow rapidly to lower redshifts. The evolution in the surface mass density profiles indicates that most of the mass at r<2 kpc was in place by z~2, and that most of the new mass growth occurred at larger radii. This inside-out mass growth is therefore responsible for the larger sizes and higher Sersic indices of the descendants toward low redshift. At z<2, the effective radius evolves with the stellar mass as r_e M^{2.0}, consistent with scenarios that find dissipationless minor mergers to be a key driver of size evolution. The progenitors at z~3 were likely star-forming disks with r_e~2 kpc, based on their low Sersic index of n~1, low median axis ratio of b/a~0.52, and typical location in the star-forming region of the U-V versus V-J diagram. By z~1.5, many of these star-forming disks disappeared, giving rise to compact quiescent galaxies. Toward lower redshifts, these galaxies continued to assemble mass at larger radii and became the local ellipticals that dominate the high mass end of the mass function at the present epoch.