Field-induced switching of ferroelectric domains with a topological vortex configuration is studied by atomic imaging and electrical biasing in an electron microscope, revealing the role of topological defects on the topologically-guided change of domain-wall pairs in a hexagonal manganite.