We present a detailed analysis of a large sample of spectroscopically confirmed ultra-massive quiescent galaxies (${rm{log}}(M_{ast}/M_{odot})sim11.5$) at $zgtrsim2$. This sample comprises 15 galaxies selected in the COSMOS and UDS fields by their bright K-band magnitudes and followed up with VLT/X-shooter spectroscopy and HST/WFC3 $H_{F160W}$ imaging. These observations allow us to unambiguously confirm their redshifts ascertain their quiescent nature and stellar ages, and to reliably assess their internal kinematics and effective radii. We find that these galaxies are compact, consistent with the high mass end of the mass-size relation for quiescent galaxies at $z=2$. Moreover, the distribution of the measured stellar velocity dispersions of the sample is consistent with the most massive local early-type galaxies from the MASSIVE Survey showing that evolution in these galaxies, is dominated by changes in size. The HST images reveal, as surprisingly high, that $40 %$ of the sample have tidal features suggestive of mergers and companions in close proximity, including three galaxies experiencing ongoing major mergers. The absence of velocity dispersion evolution from $z=2$ to $0$, coupled with a doubling of the stellar mass, with a factor of four size increase and the observed disturbed stellar morphologies support dry minor mergers as the primary drivers of the evolution of the massive quiescent galaxies over the last 10 billion years.