In this paper, we explain the dependance of the fluctuations of the largest eigenvalues of a Deformed Wigner model with respect to the eigenvectors of the perturbation matrix. We exhibit quite general situations that will give rise to universality or non universality of the fluctuations.