We utilise the final catalogue from the Pan-Andromeda Archaeological Survey to investigate the links between the globular cluster system and field halo in M31 at projected radii $R_p=25-150$ kpc. In this region the cluster radial density profile exhibits a power-law decline with index $Gamma=-2.37pm0.17$, matching that for the stellar halo component with [Fe/H] $<-1.1$. Spatial density maps reveal a striking correspondence between the most luminous substructures in the metal-poor field halo and the positions of many globular clusters. By comparing the density of metal-poor halo stars local to each cluster with the azimuthal distribution at commensurate radius, we reject the possibility of no correlation between clusters and field overdensities with high confidence. We use our stellar density measurements and previous kinematic data to demonstrate that $approx35-60%$ of clusters exhibit properties consistent with having been accreted into the outskirts of M31 at late times with their parent dwarfs. Conversely, at least $sim40%$ of remote clusters show no evidence for a link with halo substructure. The radial density profile for this subgroup is featureless and closely mirrors that observed for the apparently smooth component of the metal-poor stellar halo. We speculate that these clusters are associated with the smooth halo; if so, their properties appear consistent with a scenario where the smooth halo was built up at early times via the destruction of primitive satellites. In this picture the entire M31 globular cluster system outside $R_p=25$ kpc comprises objects accumulated from external galaxies over a Hubble time of growth.