The dramatic size evolution of early-type galaxies from z ~ 2 to 0 poses a new challenge in the theory of galaxy formation, which may not be explained by the standard picture. It is shown here that the size evolution can be explained if the non-baryonic cold dark matter is composed of compact objects having a mass scale of ~10^5 M_sun. This form of dark matter is consistent with or only weakly constrained by the currently available observations. The kinetic energy of the dark compact objects is transferred to stars by dynamical friction, and stars around the effective radius are pushed out to larger radii, resulting in a pure size evolution. This scenario has several good properties to explain the observations, including the ubiquitous nature of size evolution and faster disappearance of higher density galaxies.