In high energy heavy ion collisions, the directed flow of particles is conventionally measured with respect to that of the projectile spectators, which is defined as positive $x$ direction. But it is not known if the spectators deflect in the outward direction or inward -- toward the center line of the collision. In this Letter we discuss how the measurements of the directed flow at mid-rapidity, especially in asymmetric collision such as Cu+Au, can be used to answer this question. We show that the existing data strongly favor the case that the spectators, in the ultrarelativistic collisions, on average deflect outwards.