The aim of the project is to define a metallicity/gravity/temperature scale vs spectral types for metal-poor M dwarfs. We obtained intermediate-resolution ultraviolet (R~3300), optical (R~5400), and near-infrared (R~3900) spectra of 43 M subdwarfs (sdM), extreme subdwarfs (esdM), and ultra-subdwarfs (usdM) with the X-shooter spectrograph on the European Southern Observatory Very Large Telescope. We compared our atlas of spectra to the latest BT-Settl synthetic spectral energy distribution over a wide range of metallicities, gravities, and effective temperatures to infer the physical properties for the whole M dwarf sequence (M0--M9.5) at sub-solar metallicities and constrain the latest state-of-the-art atmospheric models. The BT-Settl models reproduce well the observed spectra across the 450-2500 nm wavelength range except for a few regions. We find that the best fits are obtained for gravities of log(g) = 5.0-5.5 dex for the three metal classes. We infer metallicities of [Fe/H] = -0.5, -1.5, and -2.0+/-0.5 dex and effective temperatures of 3700-2600 K, 3800-2900 K, and 3700-2900 K for subdwarfs, extreme subdwarfs, and ultra-subdwarfs, respectively. Metal-poor M dwarfs tend to be warmer by about 200+/-100 K and exhibit higher gravity than their solar-metallicity counterparts. We derive abundances of several elements (Fe, Na, K, Ca, Ti) for our sample but cannot describe their atmospheres with a single metallicity parameter. Our metallicity scale expands the current scales available for midly metal-poor planet-host low-mass stars. Our compendium of moderate-resolution spectra covering the 0.45--2.5 micron range represents an important legacy value for large-scale surveys and space missions to come.