We study how the charge neutrality affects the phase structure of three-flavor PNJL model. We point out that, within the conventional PNJL model at finite density the color neutrality is missing because the Wilson line serves as an external ``colored field coupled to dynamical quarks. In this paper we heuristically assume that the model may still be applicable. To get color neutrality one has then to allow non vanishing color chemical potentials. We study how the quark matter phase diagram in $(T,m_s^2/mu)$-plane is affected by imposing neutrality and by including the Polyakov loop dynamics. Although these two effects are correlated in a nonlinear way, the impact of the Polyakov loop turns out to be significant in the $T$ direction, while imposing neutrality brings a remarkable effect in the $m_s^2/mu$ direction. In particular, we find a novel unlocking transition, when the temperature is increased, even in the chiral SU(3) limit. We clarify how and why this is possible once the dynamics of the colored Polyakov loop is taken into account. Also we succeed in giving an analytic expression for $T_c$ for the transition from two-flavor pairing (2SC) to unpaired quark matter in the presence of the Polyakov loop.