The $mathbb{Z}_2$ topological phase in the quantum dimer model on the Kagome-lattice is a candidate for the description of the low-energy physics of the anti-ferromagnetic Heisenberg model on the same lattice. We study the extend of the topological phase by interpolating between the exactly solvable parent Hamiltonian of the topological phase and an effective low-energy description of the Heisenberg model in terms of a quantum-dimer Hamiltonian. Therefore, we perform a perturbative treatment of the low-energy excitations in the topological phase including free and interacting quasi-particles. We find a phase transition out of the topological phase far from the Heisenberg point. The resulting phase is characterized by a spontaneously broken rotational symmetry and a unit cell involving six sites.