It is well known that for a first order system of linear difference equations with rational function coefficients, a solution that is holomorphic in some left half plane can be analytically continued to a meromorphic solution in the whole complex plane. The poles stem from the singularities of the rational function coefficients of the system. Just as for differential equations, not all of these singularities necessarily lead to poles in solutions, as they might be what is called removable. In our work, we show how to detect and remove these singularities and further study the connection between poles of solutions and removable singularities. We describe two algorithms to (partially) desingularize a given difference system and present a characterization of removable singularities in terms of shifts of the original system.