We develop an open-system dynamical theory of the Casimir interaction between coherent atomic waves and a material surface. The system --- the external atomic waves --- disturbs the environment --- the electromagnetic field and the atomic dipole degrees of freedom --- in a non- local manner by leaving footprints on distinct paths of the atom interferometer. This induces a non-local dynamical phase depending simultaneously on two distinct paths, beyond usual atom-optics methods, and comparable to the local dynamical phase corrections. Non-local and local atomic phase coherences are thus equally important to capture the interplay between the external atomic motion and the Casimir interaction. Such dynamical phases are obtained for finite-width wavepackets by developing a diagrammatic expansion of the disturbed environment quantum state.