We investigate a model on an extra dimension $S^1$ where plenty of effective boundary points described by point interactions (zero-thickness branes) are arranged. After suitably selecting the conditions on these points for each type of five-dimensional fields, we realize the tiny active neutrino masses, the charged lepton mass hierarchy, and lepton mixings with a CP-violating phase, simultaneously. Not only the quarks but also the leptons configurations are generated in a unified way with acceptable accuracy, with neither the see-saw mechanism nor symmetries in Yukawa couplings, by suitably setting the model parameters, even though their flavor structures are dissimilar each other. One remarkable point is that a complex vacuum expectation value of the five-dimensional Higgs doublet in this model becomes the common origin of the CP violation in both quark and lepton sectors. The model can be consistent with the results of the precision electroweak measurements and Large Hadron Collider experiments.