We study the spontaneous $CP$ violation through the stabilization of the modulus $tau$ in modular invariant flavor models. The $CP$-invaraiant potentential has the minimum only at ${rm Re}[tau] = 0$ or 1/2. From this prediction, we study $CP$ violation in modular invariant flavor models. The physical $CP$ phase is vanishing. The important point for the $CP$ conservation is the $T$ transformation in the modular symmetry. One needs the violation of $T$ symmetry to realize the spontaneous $CP$ violation.