We study a flavor model with $A_4$ symmetry which originates from $S_4$ modular group. In $S_4$ symmetry, $Z_2$ subgroup can be anomalous, and then $S_4$ can be violated to $A_4$. Starting with a $S_4$ symmetric Lagrangian at the tree level, the Lagrangian at the quantum level has only $A_4$ symmetry when $Z_2$ in $S_4$ is anomalous. We obtain modular forms of two singlets and a triplet representations of $A_4$ by decomposing $S_4$ modular forms into $A_4$ representations. We propose a new $A_4$ flavor model of leptons by using those $A_4$ modular forms. We succeed in constructing a viable neutrino mass matrix through the Weinberg operator for both normal hierarchy (NH) and inverted hierarchy (IH) of neutrino masses. Our predictions of the CP violating Dirac phase $delta_{CP}$ and the mixing $sin^2theta_{23}$ depend on the sum of neutrino masses for NH.