We propose a predictive model based on the $SU(3)_Ctimes SU(3)_Ltimes U(1)_X$ gauge symmetry, which is supplemented by the $D_4$ family symmetry and several auxiliary cyclic symmetries whose spontaneous breaking produces the observed SM fermion mass and mixing pattern. The masses of the light active neutrinos are produced by an inverse seesaw mechanism mediated by three right handed Majorana neutrinos. To the best of our knowledge the model corresponds to the first implementation of the $D_4$ family symmetry in a $SU(3)_Ctimes SU(3)_Ltimes U(1)_X$ theory with three right handed Majorana neutrinos and inverse seesaw mechanism. Our proposed model successfully accommodates the experimental values of the SM fermion mass and mixing parameters, the muon anomalous magnetic moment as well as the Higgs diphoton decay rate constraints. The consistency of our model with the muon anomalous magnetic moment requires electrically charged scalar masses at the sub TeV scale.