Focused ion beam (FIB) and scanning electron microscopy (SEM) are used to reversibly switch improper ferroelectric domains in the hexagonal manganite ErMnO$_3$. Surface charging is achieved by local ion (positive charging) and electron (positive and negative charging) irradiation, which allows controlled polarization switching without the need for electrical contacts. Polarization cycling reveals that the domain walls tend to return to the equilibrium configuration obtained in the as-grown state. The electric field response of sub-surface domains is studied by FIB cross-sectioning, revealing the 3D switching behavior. The results clarify how the polarization reversal in hexagonal manganites progresses at the level of domains, resolving both domain wall movements and the nucleation and growth of new domains. Our FIB-SEM based switching approach is applicable to all ferroelectrics where a sufficiently large electric field can be built up via surface charging, facilitating contact-free high-resolution studies of the domain and domain wall response to electric fields in 3D.