F. Labourie [arXiv:1212.5015] characterized the Hitchin components for $operatorname{PSL}(n, mathbb{R})$ for any $n>1$ by using the swapping algebra, where the swapping algebra should be understood as a ring equipped with a Poisson bracket. We introduce the rank $n$ swapping algebra, which is the quotient of the swapping algebra by the $(n+1)times(n+1)$ determinant relations. The main results are the well-definedness of the rank $n$ swapping algebra and the cross-ratio in its fraction algebra. As a consequence, we use the sub fraction algebra of the rank $n$ swapping algebra generated by these cross-ratios to characterize the $operatorname{PSL}(n, mathbb{R})$ Hitchin component for a fixed $n>1$. We also show the relation between the rank $2$ swapping algebra and the cluster $mathcal{X}_{operatorname{PGL}(2,mathbb{R}),D_k}$-space.