We present the first study of the large-scale clustering of post-starburst (PSB) galaxies in the high redshift Universe ($0.5<z<3.0$). We select $sim4000$ PSB galaxies photometrically, the largest high-redshift sample of this kind, from two deep large-scale near-infrared surveys: the UKIDSS Ultra Deep Survey (UDS) DR11 and the Cosmic Evolution Survey (COSMOS). Using angular cross-correlation techniques, we estimate the halo masses for this large sample of PSB galaxies and compare them with quiescent and star-forming galaxies selected in the same fields. We find that low-mass, low-redshift ($0.5<z<1.0$) PSB galaxies preferentially reside in very high-mass dark matter halos (M$_{text{halo}}>10^{14}$M$_{odot}$), suggesting they are likely to be infalling satellite galaxies in cluster-like environments. High-mass PSB galaxies are more weakly clustered at low redshifts, but they reside in higher mass haloes with increasing look-back time, suggesting strong redshift-dependent halo downsizing. These key results are consistent with previous results suggesting that two main channels are responsible for the rapid quenching of galaxies. While high-redshift ($z>1$) galaxies appear to be quenched by secular feedback mechanisms, processes associated with dense environments are likely to be the key driver of rapid quenching in the low-redshift Universe ($z<1$). Finally, we show that the clustering of photometrically selected PSBs are consistent with them being direct descendants of highly dust-enshrouded sub-millimetre galaxies (SMGs), providing tantalising evidence for the oft-speculated evolutionary pathway from starburst to quiescence.