A significant proportion ($sim30%$) of the short-duration gamma-ray bursts (SGRBs) localised by Swift have no detected host galaxy coincident with the burst location to deep limits, and also no high-likelihood association with proximate galaxies on the sky. These SGRBs may represent a population at moderately high redshifts ($zgtrsim1$), for which the hosts are faint, or a population where the progenitor has been kicked far from its host or is sited in an outlying globular cluster. We consider the afterglow and host observations of three hostless bursts (GRBs 090305A, 091109B and 111020A), coupled with a new observational diagnostic to aid the association of SGRBs with putative host galaxies to investigate this issue. Considering the well localised SGRB sample, 7/25 SGRBs can be classified as hostless by our diagnostic. Statistically, however, the proximity of these seven SGRBs to nearby galaxies is higher than is seen for random positions on the sky. This suggests that the majority of hostless SGRBs have likely been kicked from proximate galaxies at moderate redshift. Though this result still suggests only a small proportion of SGRBs will be within the AdLIGO horizon for NS-NS or NS-BH inspiral detection ($zsim0.1$), in the particular case of GRB 111020A a plausible host candidate is at $z=0.02$.