Magnetic systems with frustration often have large classical degeneracy. We show that their low-energy physics can be understood as dynamics within the space of classical ground states. We demonstrate this mapping in a family of quantum spin clusters where every pair of spins is connected by an $XY$ antiferromagnetic bond. The dimer with two spin-$S$ spins provides the simplest example, it maps to a quantum particle on a ring ($S^1$). The trimer is more complex, equivalent to a particle that lives on two disjoint rings ($S^1otimes mathbb{Z}_2$). It has an additional subtlety for half-integer $S$ values, wherein both rings must be threaded by $pi$-fluxes to obtain a satisfactory mapping. This is a consequence of the geometric phase incurred by spins. For both the dimer and the trimer, the effective theory can be seen from a path-integral-based derivation. This approach cannot be extended to the quadrumer which has a non-manifold ground state space, consisting of three tori that touch pairwise along lines. In order to understand the dynamics of a particle in this space, we develop a tight-binding model with this connectivity. Remarkably, this successfully reproduces the low-energy spectrum of the quadrumer. For half-integer spins, a geometric phase emerges which can be mapped to two $pi$-flux tubes that reside in the space between the tori. The non-manifold character of the space leads to a remarkable effect - the dynamics at low energies is not ergodic as the particle is localized around singular lines of the ground state space. The low-energy spectrum consists of an extensive number of bound states formed around singularities. Physically, this manifests as an order-by-disorder-like preference for collinear ground states. However, unlike order-by-disorder, this `order by singularity persists even in the classical limit. We discuss consequences for field theoretic studies of magnets.