It is recently shown that in 4D $SU(N)$ Nambu--Jona-Lasinio (NJL) type models, the $SU(N)$ symmetry breaking into its special subgroups is not special but much more common than that into the regular subgroups, where the fermions belong to complex representations of $SU(N)$. We perform the same analysis for $SO(N)$ NJL model for various $N$ with fermions belonging to an irreducible spinor representation of $SO(N)$. We find that the symmetry breaking into special or regular subgroups has some correlation with the type of fermion representations; i.e., complex, real, pseudo-real representations.