We use the statistics of the VIPERS survey to investigate the relation between the surface mean stellar mass density Sigma=Mstar/(2*pi*Re^2) of massive passive galaxies (MPGs, Mstar>10^11 Msun) and their environment in the redshift range 0.5<z<0.8. Passive galaxies were selected on the basis of their NUVrK colors (~900 objects), and the environment was defined as the galaxy density contrast, delta, using the fifth nearest-neighbor approach. The analysis of Sigma vs. delta was carried out in two stellar mass bins. In galaxies with Mstar<2*10^11 Msun, no correlation between Sigma and delta is observed. This implies that the accretion of satellite galaxies, which is more frequent in denser environments and efficient in reducing the galaxy Sigma, is not relevant in the formation and evolution of these systems. Conversely, in galaxies with Mstar>2*10^11 Msun, we find an excess of MPGs with low Sigma and a deficit of high-Sigma MPGs in the densest regions wrt other environments. We interpret this result as due to the migration of some high-Sigma MPGs (<1% of the total population of MPGs) into low-Sigma MPGs, probably through mergers or cannibalism of small satellites. In summary, our results imply that the accretion of satellite galaxies has a marginal role in the mass-assembly history of most MPGs. We have previously found that the number density of VIPERS massive star-forming galaxies (MSFGs) declines rapidily from z=0.8 to z=0.5, which mirrors the rapid increase in the number density of MPGs. This indicates that the MSFGs at z>0.8 migrate to the MPG population. Here, we investigate the Sigma-delta relation of MSFGs at z>0.8 and find that it is consistent within 1 sigma with that of low-Sigma MPGs at z<0.8. Thus, the results of this and our previous paper show that MSFGs at z>0.8 are consistent in terms of number and environment with being the progenitors of low-Sigma MPGs at z<0.8.