Deterministic polarization reversal in ferroelectric and multiferroic films is critical for their exploitation in nanoelectronic devices. While ferroelectricity has been studied for nearly a century, major discrepancies in the reported values of coercive fields and saturation polarization persist in literature for many materials. This raises questions about the atomic-scale mechanisms behind polarization reversal. Unconventional ferroelectric switching in $epsilon$-Fe2O3 films, a material that combines ferrimagnetism and ferroelectricity at room temperature, is reported here. High-resolution in-situ scanning transmission electron microscopy (STEM) experiments and first-principles calculations demonstrate that polarization reversal in $epsilon$-Fe2O3 occurs around pre-existing domain walls only, triggering local domain wall motion in moderate electric fields of 250 - 500 kV/cm. Calculations indicate that the activation barrier for switching at domain walls is nearly a quarter of that corresponding to the most likely transition paths inside $epsilon$-Fe2O3 domains. Moreover, domain walls provide symmetry lowering, which is shown to be necessary for ferroelectric switching. Local polarization reversal in $epsilon$-Fe2O3 limits the macroscopic ferroelectric response and offers important hints on how to tailor ferroelectric properties by domain structure design in other relevant ferroelectric materials.