Dissipationless (gas-free or dry) mergers have been suggested to play a major role in the formation and evolution of early-type galaxies, particularly in growing their mass and size without altering their stellar populations. We perform a new test of the dry merger hypothesis by comparing N-body simulations of realistic systems to empirical constraints provided by recent studies of lens early-type galaxies. We find that major and minor dry mergers: i) preserve the nearly isothermal structure of early-type galaxies within the observed scatter; ii) do not change more than the observed scatter the ratio between total mass M and virial mass R_e*sigma/2G (where R_e is the half-light radius and sigma the projected velocity dispersion); iii) increase strongly galaxy sizes [as M^(0.85+/-0.17)] and weakly velocity dispersions [as M^(0.06+/-0.08)] with mass, thus moving galaxies away from the local observed M-R_e and M-sigma relations; iv) introduce substantial scatter in the M-R_e and M-sigma relations. Our findings imply that, unless there is a high degree of fine tuning of the mix of progenitors and types of interactions, present-day massive early-type galaxies cannot have assembled more than ~50% of their mass, and increased their size by more than a factor ~1.8, via dry merging.