In this paper, we derive the second variation formula of pseudoharmonic maps into any pseudo-Hermitian manifolds. When the target manifold is an isometric embedded CR manifold in complex Euclidean space or a pseudo-Hermitian immersed submanifold in Heisenberg group, we give some conditions on Weingarten maps to obtain some unstability of pseudoharmonic maps between these pseudo-Hermitian manifolds.