We study the realization of anyon-permuting symmetries of topological phases on the lattice using tensor networks. Working on the virtual level of a projected entangled pair state, we find matrix product operators (MPOs) that realize all unitary topological symmetries for the toric and color codes. These operators act as domain walls that enact the symmetry transformation on anyons as they cross. By considering open boundary conditions for these domain wall MPOs, we show how to introduce symmetry twists and defect lines into the state.