Relativistic nuclear collisions offer a unique way to study strong interactions at very high energy. The collision process can be described within the gluon saturation framework as the interaction of two colored glasses, and because of this interaction strong longitudinal gluon fields, namely the Glasma, are produced immediately after the collision. Besides, heavy quarks are also produced in the very early stage and because of their large mass and small concentration, their motion does not affect the evolution of the Glasma, thus behaving as ideal probes of the Glasma itself. We study the evolution of the heavy quarks in the Glasma allegedly produced in high energy p-Pb collisions by solving consistently the equations of motion of the quarks in the evolving Glasma fields. We find that this motion can be understood in terms of diffusion in momentum space, similarly to the random motion of a heavy probe in a hot thermalized medium. We show how the diffusion of heavy probes affects the nuclear modification factor of D and B mesons in p-Pb collisions.