It has been known that the centralizer $Z_W(W_I)$ of a parabolic subgroup $W_I$ of a Coxeter group $W$ is a split extension of a naturally defined reflection subgroup by a subgroup defined by a 2-cell complex $mathcal{Y}$. In this paper, we study the structure of $Z_W(W_I)$ further and show that, if $I$ has no irreducible components of type $A_n$ with $2 leq n < infty$, then every element of finite irreducible components of the inner factor is fixed by a natural action of the fundamental group of $mathcal{Y}$. This property has an application to the isomorphism problem in Coxeter groups.