We develop a novel and unifying setting for phase retrieval problems that works in Banach spaces and for continuous frames and consider the questions of uniqueness and stability of the reconstruction from phaseless measurements. Our main result states that also in this framework, the problem of phase retrieval is never uniformly stable in infinite dimensions. On the other hand, we show weak stability of the problem. This complements recent work [9], where it has been shown that phase retrieval is always unstable for the setting of discrete frames in Hilbert spaces. In particular, our result implies that the stability properties cannot be improved by oversampling the underlying discrete frame. We generalize the notion of complement property (CP) to the setting of continuous frames for Banach spaces (over $mathbb{K}=mathbb{R}$ or $mathbb{K}=mathbb{C}$) and verify that it is a necessary condition for uniqueness of the phase retrieval problem; when $mathbb{K}=mathbb{R}$ the CP is also sufficient for uniqueness. In our general setting, we also prove a conjecture posed by Bandeira et al. [5], which was originally formulated for finite-dimensional spaces: for the case $mathbb{K}=mathbb{C}$ the strong complement property (SCP) is a necessary condition for stability. To prove our main result, we show that the SCP can never hold for frames of infinite-dimensional Banach spaces.