We report an anticorrelation between continuum luminosity and the equivalent width (EW) of the H-alpha emission line in X-ray binary systems. The effect is evident both in a universal monotonic increase in H-alpha EW with time following outbursts, as systems fade, and in a comparison between measured EWs and contemporaneous X-ray measurements. The effect is most clear for black hole binaries in the low/hard X-ray state, which is prevalent at X-ray luminosities below ~1% Eddington. We do not find strong evidence for significant changes in line profiles across accretion state changes, but this is hampered by a lack of good data at such times. The observed anti-correlation, highly significant for black hole binaries, is only marginally so for neutron star systems, for which there are far less data. Comparison with previously established correlations between optical and X-ray luminosity suggest that the line luminosity is falling as the X-ray and optical luminosities drop, but not as fast (approximately as L_{H-alpha} propto L_X^{~0.4} propto L_{opt}^{~0.7}). We briefly discuss possible origins for such an effect, including the optical depth, form of the irradiating spectrum and geometry of the accetion flow. Further refinement of the relation in the future may allow measurements of H-alpha EW to be used to estimate the luminosity of, and hence the distance to, X-ray binary systems. Beyond this, further progress will require a better sample of spectro-photometric data.