The problem of reconstructing a function from the magnitudes of its frame coefficients has recently been shown to be never uniformly stable in infinite-dimensional spaces [5]. This result also holds for frames that are possibly continuous [2]. On the other hand, the problem is always stable in finite-dimensional settings. A prominent example of such a phase retrieval problem is the recovery of a signal from the modulus of its Gabor transform. In this paper, we study Gabor phase retrieval and ask how the stability degrades on a natural family of finite-dimensional subspaces of the signal domain $L^2(mathbb{R})$. We prove that the stability constant scales at least quadratically exponentially in the dimension of the subspaces. Our construction also shows that typical priors such as sparsity or smoothness promoting penalties do not constitute regularization terms for phase retrieval.