Using a mechanism which allows naturally small Dirac neutrino masses and its linkage to a dark gauge $U(1)_D$ symmetry, a realistic Dirac neutrino mass matrix is derived from $S_3$. The dark sector naturally contains a fermion singlet having a small seesaw mass. It is thus a good candidate for freeze-in dark matter from the decay of the $U(1)_D$ Higgs boson.