We study lepton flavor models with the $S_4$ flavor symmetry. We construct simple models with smaller numbers of flavon fields and free parameters, such that we have predictions among lepton masses and mixing angles. The model with a $S_4$ triplet flavon is not realistic, but we can construct realistic models with two triplet flavons, or one triplet and one doublet flavons.