The Lyman-$alpha$ forest is a powerful tool to constrain warm dark matter models (WDM). Its main observable -- flux power spectrum -- should exhibit a suppression at small scales in WDM models. This suppression, however, can be mimicked by a number of thermal effects related to the instantaneous temperature of the intergalactic medium (IGM), and to the history of reionization and of the IGM heating (pressure effects). Therefore, to put robust bounds on WDM one needs to disentangle the effect of free-streaming of dark matter particles from the influence of all astrophysical effects. This task cannot be brute-forced due to the complexity of the IGM modelling. In this work, we model the sample of high-resolution and high-redshift quasar spectra (Boera et al 2018) assuming a thermal history that leads to the smallest pressure effects while still being broadly compatible with observations. We explicitly marginalize over observationally allowed values of IGM temperature and find that (thermal) WDM models with masses above 1.9 keV (at 95% CL) are consistent with the spatial shape of the observed flux power spectrum at $z=4-5$. Even warmer models would produce a suppression at scales that are larger than observed, independently of assumptions about thermal effects. This bound is significantly lower than previously claimed bounds, demonstrating the importance of the knowledge about the reionization history and of the proper marginalization over unknowns.