We present a detailed analysis of the Galaxy Stellar Mass Function of galaxies up to z=2.5 as obtained from the VVDS. We estimate the stellar mass from broad-band photometry using 2 different assumptions on the galaxy star formation history and show that the addition of secondary bursts to a continuous star formation history produces systematically higher (up to 40%) stellar masses. At low redshift (z=0.2) we find a substantial population of low-mass galaxies (<10^9 Msun) composed by faint blue galaxies (M_I-M_K=0.3). In general the stellar mass function evolves slowly up to z=0.9 and more significantly above this redshift. Conversely, a massive tail is present up to z=2.5 and have extremely red colours (M_I-M_K=0.7-0.8). We find a decline with redshift of the overall number density of galaxies for all masses (59+-5% for M>10^8 Msun at z=1), and a mild mass-dependent average evolution (`mass-downsizing). In particular our data are consistent with mild/negligible (<30%) evolution up to z=0.7 for massive galaxies (>6x10^10 Msun). For less massive systems the no-evolution scenario is excluded. A large fraction (>=50%) of massive galaxies have been already assembled and converted most of their gas into stars at z=1, ruling out the `dry mergers as the major mechanism of their assembly history below z=1. This fraction decreases to 33% at z=2. Low-mass systems have decreased continuously in number and mass density (by a factor up to 4) from the present age to z=2, consistently with a prolonged mass assembly also at z<1.