We present a treatment of decoherence in an atom due to scattering from a gas of free particles. We show that there is a recoil free scattering process that leaves both the atom and the gas in an unchanged state, but allows for the acquisition of a phase shift that remains in the free space limit. This is essential to understanding decoherence in a separated arm atom interferometer, where a gas of atoms forms a refractive medium for a matter wave. Our work clarifies the extent to which scattering of a free particle acts as a which-way measurement.