We present deep VLT and HST observations of the nearest examples of calcium-rich gap transients -- rapidly evolving transient events, with a luminosity intermediate between novae and supernovae. These sources are frequently found at large galactocentric offsets, and their progenitors remain mysterious. Our observations find no convincing underlying quiescent sources coincident with the locations of these transients, allowing us to rule out a number of potential progenitor systems. The presence of surviving massive-star binary companions (or other cluster members) are ruled out, providing an independent rejection of a massive star origin for these events. Dwarf satellite galaxies are disfavoured unless one invokes as yet unknown conditions that would be extremely favourable for their production in the lowest mass systems. Our limits also probe the majority of the globular cluster luminosity function, ruling out the presence of an underlying globular cluster population at high significance, and thus the possibility that they are created via dynamical interactions in dense globular cluster cores. Given the lack of underlying systems, previous progenitor suggestions have difficulty reproducing the remote locations of these transients, even when considering solely halo-borne progenitors. Our preferred scenario is that calcium-rich transients are high velocity, kicked systems, exploding at large distances from their natal site. Coupled with a long-lived progenitor system post-kick, this naturally explains the lack of association these transients have with their host stellar light, and the extreme host-offsets exhibited. Neutron star -- white dwarf mergers may be a promising progenitor system in this scenario.