We study the modulus stabilization in an $A_4$ model whose $A_4$ flavor symmetry is originated from the $S_4$ modular symmetry. We can stabilize the modulus so that the $A_4$ invariant superpotential leads to the realistic lepton masses and mixing angles. We also discuss the phenomenological aspect of the present model as a consequence of the modulus stabilization.