Primordial black holes as dark matter may be generated in single-field models of inflation thanks to the enhancement at small scales of the comoving curvature perturbation. This mechanism requires leaving the slow-roll phase to enter a non-attractor phase during which the inflaton travels across a plateau and its velocity drops down exponentially. We argue that quantum diffusion has a significant impact on the primordial black hole mass fraction making the classical standard prediction not trustable.