We consider solutions of the matrix KP hierarchy that are elliptic functions of the first hierarchical time $t_1=x$. It is known that poles $x_i$ and matrix residues at the poles $rho_i^{alpha beta}=a_i^{alpha}b_i^{beta}$ of such solutions as functions of the time $t_2$ move as particles of spin generalization of the elliptic Calogero-Moser model (elliptic Gibbons-Hermsen model). In this paper we establish the correspondence with the spin elliptic Calogero-Moser model for the whole matrix KP hierarchy. Namely, we show that the dynamics of poles and matrix residues of the solutions with respect to the $k$-th hierarchical time of the matrix KP hierarchy is Hamiltonian with the Hamiltonian $H_k$ obtained via an expansion of the spectral curve near the marked points. The Hamiltonians are identified with the Hamiltonians of the elliptic spin Calogero-Moser system with coordinates $x_i$ and spin degrees of freedom $a_i^{alpha}, , b_i^{beta}$.