We identify the dimension of the centralizer of the symmetric group $mathfrak{S}_d$ in the partition algebra $mathcal{A}_d(delta)$ and in the Brauer algebra $mathcal{B}_d(delta)$ with the number of multidigraphs with $d$ arrows and the number of disjoint union of directed cycles with $d$ arrows, respectively. Using Schur-Weyl duality as a fundamental theory, we conclude that each centralizer is related with the $G$-invariant space $P^d(M_n(mathbf{k}))^G$ of degree $d$ homogeneous polynomials on $n times n$ matrices, where $G$ is the orthogonal group and the group of permutation matrices, respectively. Our approach gives a uniform way to show that the dimensions of $P^d(M_n(mathbf{k}))^G$ are stable for sufficiently large $n$.