We develop a self-avoiding walks approximation and use it to calculate the spectral function of a single hole introduced into the 2D square lattice Ising antiferromagnet. The obtained local spectral function qualitatively agrees with the exact diagonalisation result and is largely incoherent. Such a result stays in contrast with the spectrum obtained on a Bethe lattice, which consists of the well-separated quasiparticle-like peaks and stems from the motion of a hole in an effective linear potential. We determine that this onset of the incoherent spectrum on a square lattice (i) is not triggered by the so-called Trugman loops but (ii) originates in the warping of the linear potential by the interactions between magnons created along the tangential paths of the moving hole.