Star formation is spatially clustered across a range of environments, from dense stellar clusters to unbound associations. As a result, radiative or dynamical interactions with neighbouring stars disrupt (proto)planetary systems and limit their radii, leaving a lasting impact on their potential habitability. In the solar neighbourhood, we find that the vast majority of stars form in unbound associations, such that the interaction of (proto)planetary systems with neighbouring stars is limited to the densest sub-regions. However, the fraction of star formation occurring in compact clusters was considerably higher in the past, peaking at ~50% in the young Milky Way at redshift z~2. These results demonstrate that the large-scale star formation environment affects the demographics of planetary systems and the occupation of the habitable zone. We show that planet formation is governed by multi-scale physics, in which Mpc-scale events such as galaxy mergers affect the AU-scale properties of (proto)planetary systems.