We study the CP-violating phase of the quark sector in the $U(8)$ flavor model on $T^2/Z_N , (N=2,3,4,6)$ with non-vanishing magnetic fluxes, where properties of possible origins of the CP violation are investigated minutely. In this system, a non-vanishing value is mandatory in the real part of the complex modulus parameter $tau$ of the two-dimensional torus. On $T^2$ without orbifolding, underlying discrete flavor symmetries severely restrict the form of Yukawa couplings and it is very difficult to reproduce the observed pattern in the quark sector including the CP-violating phase $delta_{rm CP}$. In cases of multiple Higgs doublets emerging on $T^2/Z_2$, the mass matrices of the zero-mode fermions can be written in the Gaussian textures by choosing appropriate configurations of vacuum expectation values of the Higgs fields. When such Gaussian textures of mass matrices are realized, we show that all of the quark profiles, which are mass hierarchies among the quarks, quark mixing angles, and $delta_{rm CP}$ can be simultaneously realized.