In this paper, we suggest a simple model which induces realistic flavor structure from mixing of flavored Higgs doublets. The idea is based on the decoupling limit. In a model with many Higgs doublets, the mass eigenstates of scalars are linear combinations of Higgs doublets. If the mass matrix of Higgs fields has only one massless mode, and if the linear combination has flavor dependence, the induced Yukawa coupling will have nontrivial flavor structure. We suggest a mass matrix of flavored Higgs fields in a $U(2)_{L} times U(2)_{R}$ toy model. An advantage of this model is that all of the elements in Yukawa matrix can be determined from renormalizable Higgs potential.