We study the structural origin of the Bauschinger effect by accessing numerically the local plastic thresholds in the steady state flow of a two-dimensional model glass under athermal quasistatic deformation. More specifically, we compute the local residual strength, $Deltatau^{c}$, for arbitrary loading orientations and find that plastic deformation generically induces material polarization, i.e., a forward-backward asymmetry in the $Deltatau^{c}$ distribution. In steady plastic flow, local packings are on average closer to forward (rather than backward) instabilities, due to the stress-induced bias of barriers. However, presumably due to mechanical noise, a significant fraction of zones lie close to reverse (backward) yielding, as the distribution of $Deltatau^{c}$ for reverse shearing extends quasilinearly down to zero local residual strength. By constructing an elementary model of the early plastic response, we then show that unloading causes reverse plasticity of a growing amplitude, i.e., reverse softening, while it shifts away forward-yielding barriers. This result in an inversion of polarization in the low-$Deltatau^{c}$ region and, consequently, in the Bauschinger effect. This scenario is quite generic, which explains the pervasiveness of the effect.