In this paper, we investigate the geometric phase of the field interacting with $Xi $-type moving three-level atom. The results show that the atomic motion and the field-mode structure play important roles in the evolution of the system dynamics and geometric phase. We test this observation with experimentally accessible parameters and some new aspects are obtained.