We study the possibility for a unitary theory of partially-massless (PM) spin-two field interacting with Gravity in arbitrary dimensions. We show that the gauge and parity invariant interaction of PM spin two particles requires the inclusion of specific massive spin-two fields and leads to a reconstruction of Conformal Gravity, or multiple copies of the latter in even dimensions. By relaxing the parity invariance, we find a possibility of a unitary theory in four dimensions, but this theory cannot be constructed in the standard formulation, due to the absence of the parity-odd cubic vertex therein. Finally, by relaxing the general covariance, we show that a `non-geometric coupling between massless and PM spin-two fields may lead to an alternative possibility of a unitary theory. We also clarify some aspects of interactions between massless, partially-massless and massive fields, and resolve disagreements in the literature.