We investigate the lattice ${mathbb C} P^{N-1}$ sigma model on $S_{s}^{1}$(large) $times$ $S_{tau}^{1}$(small) with the ${mathbb Z}_{N}$ symmetric twisted boundary condition, where a sufficiently large ratio of the circumferences ($L_{s}gg L_{tau}$) is taken to approximate ${mathbb R} times S^1$. We find that the expectation value of the Polyakov loop, which is an order parameter of the ${mathbb Z}_N$ symmetry, remains consistent with zero ($|langle Prangle|sim 0$) from small to relatively large inverse coupling $beta$ (from large to small $L_{tau}$). As $beta$ increases, the distribution of the Polyakov loop on the complex plane, which concentrates around the origin for small $beta$, isotropically spreads and forms a regular $N$-sided-polygon shape (e.g. pentagon for $N=5$), leading to $|langle Prangle| sim 0$. By investigating the dependence of the Polyakov loop on $S_{s}^{1}$ direction, we also verify the existence of fractional instantons and bions, which cause tunneling transition between the classical $N$ vacua and stabilize the ${mathbb Z}_{N}$ symmetry. Even for quite high $beta$, we find that a regular-polygon shape of the Polyakov-loop distribution, even if it is broken, tends to be restored and $|langle Prangle|$ gets smaller as the number of samples increases. To discuss the adiabatic continuity of the vacuum structure from another viewpoint, we calculate the $beta$ dependence of ``pseudo-entropy density $proptolangle T_{xx}-T_{tautau}rangle$. The result is consistent with the absence of a phase transition between large and small $beta$ regions.