We provide strong evidence that all tree-level 4-point holographic correlators in AdS$_3 times S^3$ are constrained by a hidden 6D conformal symmetry. This property has been discovered in the AdS$_5 times S^5$ context and noticed in the tensor multiplet subsector of the AdS$_3 times S^3$ theory. Here we extend it to general AdS$_3 times S^3$ correlators which contain also the chiral primary operators of spin zero and one that sit in the gravity multiplet. The key observation is that the 6D conformal primary field associated with these operators is not a scalar but a self-dual $3$-form primary. As an example, we focus on the correlators involving two fields in the tensor multiplets and two in the gravity multiplet and show that all such correlators are encoded in a conformal 6D correlator between two scalars and two self-dual $3$-forms, which is determined by three functions of the cross ratios. We fix these three functions by comparing with the results of the simplest correlators derived from an explicit supergravity calculation.