We discuss the relation between the CP violation of the quark mixing and that of the lepton mixing by investigating a CP violating observable, the Jarlskog invariant, as well as the CP violating Dirac phase. The down-type quark mass matrix with three zeros is given in terms of the minimal number of parameters, while the up-type quark mass matrix is diagonal. These quark mass matrices leading to the successful CKM mixing angles and CP violation are embedded in both the Pati--Salam and SU(5) models. The leptonic Jarlskog invariant $J_{CP}^l$ (as well as CP violating Dirac phase) is examined for two cases: the neutrino mass matrix is diagonal or non-diagonal, where no additional CP violating phase is introduced apart from the Majorana phases. In the case of the diagonal neutrino mass matrix, the favorable sign of the leptonic CP violation is obtained, however, the magnitude of $J_{CP}^l$ is at most ${cal O}(10^{-4})$, which is too small compared with the expected value from the observation $-0.02$. In the case of the non-diagonal neutrino mass matrix where the tri-bimaximal mixing pattern is taken, we obtain the successful $J_{CP}^l$ up to its sign.