Let $Gamma$ denote a $Q$-polynomial distance-regular graph with vertex set $X$ and diameter $D$. Let $A$ denote the adjacency matrix of $Gamma$. Fix a base vertex $xin X$ and for $0 leq i leq D$ let $E^*_i=E^*_i(x)$ denote the projection matrix to the $i$th subconstituent space of $Gamma$ with respect to $x$. The Terwilliger algebra $T(x)$ of $Gamma$ with respect to $x$ is the semisimple subalgebra of $mathrm{Mat}_X(mathbb{C})$ generated by $A, E^*_0, E^*_1, ldots, E^*_D$. Remark that the isomorphism class of $T(x)$ depends on the choice of the base vertex $x$. We say $Gamma$ is pseudo-vertex-transitive whenever for any vertices $x,y in X$, the Terwilliger algebras $T(x)$ and $T(y)$ are isomorphic. In this paper we discuss pseudo-vertex transitivity for distance-regular graphs with diameter $Din {2,3,4}$. In the case of diameter $2$, a strongly regular graph $Gamma$ is thin, and $Gamma$ is pseudo-vertex-transitive if and only if every local graph of $Gamma$ has the same spectrum. In the case of diameter $3$, Taylor graphs are thin and pseudo-vertex-transitive. In the case of diameter $4$, antipodal tight graphs are thin and pseudo-vertex-transitive.