We investigate the evolution of the galaxy stellar mass function at high-redshift ($zge 5$) using a pair of large cosmological hydrodynamical simulations: {em MassiveBlack} and {em MassiveBlack-II}. By combining these simulations we can study the properties of galaxies with stellar masses greater than $10^{8},{rm M_{odot}},h^{-1}$ and (co-moving) number densities of $log_{10}(phi, [{rm Mpc^{-3},dex^{-1}},h^{3}])>-8$. Observational determinations of the galaxy stellar mass function at very-high redshift typically assume a relation between the observed UV luminosity and stellar mass-to-light ratio which is applied to high-redshift samples in order to estimate stellar masses. This relation can also be measured from the simulations. We do this, finding two significant differences with the usual observational assumption: it evolves strongly with redshift and has a different shape. Using this relation to make a consistent comparison between galaxy stellar mass functions we find that at $z=6$ and above the simulation predictions are in good agreement with observed data over the whole mass range. Without using the correct UV luminosity and stellar mass-to-light ratio, the discrepancy would be up to two orders of magnitude for large galaxies $>10^{10},{rm M_{odot}},h^{-1}$. At $z=5$, however the stellar mass function for low mass $<10^{9},{rm M_{odot}},h^{-1}$ galaxies is overpredicted by factors of a few, consistent with the behaviour of the UV luminosity function, and perhaps a sign that feedback in the simulation is not efficient enough for these galaxies.