The middle-aged PSR J0357+3205 is a nearby, radio-quiet, bright gamma-ray pulsar discovered by the Fermi mission. Our previous Chandra observation revealed a huge, very peculiar structure of diffuse X-ray emission, originating at the pulsar position and extending for > 9 on the plane of the sky. To better understand the nature of such a nebula, we have studied the proper motion of the parent pulsar. We performed relative astrometry on Chandra images of the field spanning a time baseline of 2.2 yr, unveiling a significant angular displacement of the pulsar counterpart, corresponding to a proper motion of 0.165+/-0.030 yr^(-1). At a distance of ~500 pc, the space velocity of the pulsar would be of ~390 km s^(-1) assuming no inclination with respect to the plane of the sky. The direction of the pulsar proper motion is perfectly aligned with the main axis of the X-ray nebula, pointing to a physical, yet elusive link between the nebula and the pulsar space velocity. No optical emission in the H_alpha line is seen in a deep image collected at the Gemini telescope, which implies that the interstellar medium into which the pulsar is moving is fully ionized.