We study the minimal seesaw model, where two right-handed Majorana neutrinos are introduced, focusing on the CP violating phase. In addition, we take the trimaximal mixing pattern for the neutrino flavor where the charged lepton mass matrix is diagonal. Thanks to this symmetric framework, the $3times 2$ Dirac neutrino mass matrix is given in terms of a few parameters. Numerical studies reveal that the observation of the CP violating phase can determine the flavor structure of the Dirac neutrino mass matrix in the minimal seesaw model. In particular, new minimal Dirac neutrino mass matrices are proposed in the case of $rm TM_1$, which is derived by the additional 2-3 family mixing to the tri-bimaximal mixing basis in the normal hierarchy of neutrino masses. Our analyses include the Littlest seesaw model by King {it et al.}, which is one of the specific one in our results. Furthermore, it is remarked that our $3times 2$ Dirac neutrino mass matrix is reproduced by introducing gauge singlet flavons with the specific alignments of the VEVs. These alignments suggest the residual symmetry of $S_4$ group.