We use the number counts of X-ray selected normal galaxies to explore their evolution by combining the most recent wide-angle shallow and pencil-beam deep samples available. The differential X-ray number counts, dN/dS, for early and late-type normal galaxies are constructed separately and then compared with the predictions of the local X-ray luminosity function under different evolution scenarios. The dN/dS of early type galaxies is consistent with no evolution out to z~0.5. For late-type galaxies our analysis suggests that it is the sources with X-ray--to--optical flux ratio logfx/fopt>-2 that are evolving the fastest. Including these systems in the late-type galaxy sample yields evolution of the form ~(1+z)^{2.7} out to z~0.4. On the contrary late-type sources with logfx/fopt<-2 are consistent with no evolution. This suggests that the logfx/fopt>-2 population comprises the most powerful and fast evolving starbursts at moderate and high-z. We argue that although residual low-luminosity AGN contamination may bias our results toward stronger evolution, this is unlikely to modify our main conclusions.