The hard to soft state transition of the outbursts in X-ray binaries (XRBs) is triggered by the rising of the mass accretion rate due to the disk instability. In order to explain the observed correlation between the hard X-ray transition luminosity and the soft X-ray peak luminosity in the soft state, we construct a magnetic disk-outflow model for the state transition in XRBs. We assume that the large-scale magnetic field in the outer thin disk is formed through inverse cascade of small-scale dynamo generated field, and it is then advected by the inner advection dominated accretion flow (ADAF), which accelerates a fraction of the gas into the outflows. During the outbursts, the heating front moves inwards, and the field strength at the heating front of the outer disk is proportional to the accretion rate of the disk. Much angular momentum of the inner ADAF is carried away by the outflows for a stronger magnetic field, which leads to a high radial velocity of the ADAF. This makes the critical mass accretion rate of the ADAF increases with the field strength, and it therefore leads to a correlation between transition luminosity and the peak luminosity in the thermal state. We found that the values of the viscosity parameter $alpha$ of the neutron star XRBs are systematically higher for those of the black hole (BH) XRBs ($alphasim 0.05-0.15$ for BHs, and $alphasim 0.15-0.4$ for neutron stars). Our model predicts the transition luminosity may be higher than the peak luminosity provided $alpha$ is sufficiently high, which is able to explain a substantial fraction of outbursts in BHXRBs not reaching the thermally dominant accretion state.