We present a model that gives a natural explanation to the charged lepton mass hierarchy and study the contributions to the electron and the muon $g-2$. In the model, we introduce lepton-flavor-dependent $U(1)_F$ symmetry and three additional Higgs doublets with $U(1)_F$ charges, to realize that each generation of charged leptons couples to one of the three additional Higgs doublets. The $U(1)_F$ symmetry is softly broken by $+1$ charges, and the smallness of the soft breaking naturally gives rise to the hierarchy of the Higgs VEVs, which then accounts for the charged lepton mass hierarchy. Since electron and muon couple to different scalar particles, each scalar contributes to the electron and the muon $g-2$ differently. We survey the space of parameters of the Higgs sector and find that there are sets of parameters that explain the muon $g-2$ discrepancy. On the other hand, we cannot find the parameter sets that can explain $g-2$ discrepancy within 2$sigma$. Here the $U(1)_F$ symmetry suppresses charged lepton flavor violation.