We investigate how observations of the lepton flavor violating decay of the Higgs boson ($h to ellell^prime$) can narrow down models of neutrino mass generation mechanisms, which were systematically studied in Refs. [1,2] by focusing on the combination of new Yukawa coupling matrices with leptons. We find that a wide class of models for neutrino masses can be excluded if evidence for $h to ellell^prime$ is really obtained in the current or future collider experiments. In particular, simple models of Majorana neutrino masses cannot be compatible with the observation of $h to ellell^prime$. It is also found that some of the simple models to generate masses of Dirac neutrinos radiatively can be compatible with a significant rate of the $h to ellell^prime$ process.