Minor mergers are thought to be responsible for the size growth of quiescent field galaxies with decreasing redshift. We test this hypothesis using the cluster environment as a laboratory. Satellite galaxies in clusters move at high velocities, making mergers between them rare. The stellar mass-size relation in ten clusters and in the field is measured and compared at $z~mathtt{sim}~1$. Our cluster sample contains 344 spectroscopically-confirmed cluster members with Gemini/GMOS and 182 confirmed with HST WFC3 G141 grism spectroscopy. On average, quiescent and star-forming cluster galaxies are smaller than their field counterparts by ($0.08pm0.04$) dex and ($0.07pm0.01$) dex respectively. These size offsets are consistent with the average sizes of quiescent and star-forming field galaxies between $1.2leqslant zleqslant1.5$, implying the cluster environment has inhibited size growth between this period and $z~mathtt{sim}~1$. The negligible differences measured between the $z~mathtt{sim}~0$ field and cluster quiescent mass-size relations in other works imply that the average size of quiescent cluster galaxies must rise with decreasing redshift. Using a toy model, we show that the disappearance of the compact cluster galaxies might be explained if, on average, $mathtt{sim}40%$ of them merge with their brightest cluster galaxies (BCGs) and $mathtt{sim}60%$ are tidally destroyed into the intra-cluster light (ICL) between $0leqslant zleqslant1$. This is in agreement with the observed stellar mass growth of BCGs between $0leqslant zleqslant1$ and the observed ICL stellar mass fraction at $z~mathtt{sim}~0$. Our results support minor mergers as the cause for the size growth in quiescent field galaxies, with cluster-specific processes responsible for the similarity between the field and cluster quiescent mass-size relations at low redshift.