We propose new backgrounds of extra dimensions to lead to four-dimensional chiral models with three generations of matter fermions, that is $T^2/Z_N$ twisted orbifolds with magnetic fluxes. We consider gauge theory on six-dimensional space-time, which contains the $T^2/Z_N$ orbifold with magnetic flux, Scherk-Schwarz phases and Wilson line phases. We classify all the possible Scherk-Schwarz and Wilson line phases on $T^2/Z_N$ orbifolds with magnetic fluxes. The behavior of zero modes is studied. We derive the number of zero modes for each eigenvalue of the $Z_N$ twist, showing explicitly examples of wave functions. We also investigate Kaluza-Klein mode functions and mass spectra.