Using deep Hubble Frontier Fields imaging and slitless spectroscopy from the Grism Lens-Amplified Survey from Space, we analyze 2200 cluster and 1748 field galaxies at $0.2leq zleq0.7$ to determine the impact of environment on galaxy size and structure at $log M_*/M_odot>7.8$, an unprecedented limit at these redshifts. Based on simple assumptions-$r_e=f(M_*)$-we find no significant differences in half-light radii ($r_e$) between equal-mass cluster or field systems. More complex analyses-$r_e=f(M_*,U-V,n,z,Sigma$)-reveal local density $(Sigma$) to induce only a $7% pm 3%$ ($95%$ confidence) reduction in $r_e$ beyond what can be accounted for by $U-V$ color, Sersic index ($n$), and redshift ($z$) effects.Almost any size difference between galaxies in high- and low-density regions is thus attributable to their different distributions in properties other than environment. Indeed, we find a clear color-$r_e$ correlation in low-mass passive cluster galaxies ($log M_*/M_odot<9.8$) such that bluer systems have larger radii, with the bluest having sizes consistent with equal-mass star-forming galaxies. We take this as evidence that large-$r_e$ low-mass passive cluster galaxies are recently acquired systems that have been environmentally quenched without significant structural transformation (e.g., by ram pressure stripping or starvation).Conversely, $sim20%$ of small-$r_e$ low-mass passive cluster galaxies appear to have been in place since $zsim3$. Given the consistency of the small-$r_e$ galaxies stellar surface densities (and even colors) with those of systems more than ten times as massive, our findings suggest that clusters mark places where galaxy evolution is accelerated for an ancient base population spanning most masses, with late-time additions quenched by environment-specific mechanisms are mainly restricted to the lowest masses.