We show that the jet power $P_j$ and geometrically corrected $gamma$-ray luminosity $L_gamma$ for the X-ray binaries (XRBs) Cygnus X-1, Cygnus X-3, and V404 Cygni, and $gamma$-ray upper limits for GRS 1915+105 and GX339-4, follow the universal scaling for the energetics of relativistic jets from black hole (BH) systems found by Nemmen et al. (2012) for blazars and GRBs. The observed peak $gamma$-ray luminosity for XRBs is geometrically corrected; and the minimum jet power is estimated from the peak flux density of radio flares and the flare rise time. The $L_gamma-P_j$ correlation holds across $sim 17$ orders of magnitude. The correlation suggests a jet origin for the high energy emission from X-ray binaries, and indicates a common mechanism or efficiency for the high energy emission 0.1-100 GeV from all relativistic BH systems.