We consider solutions of the KP hierarchy which are elliptic functions of $x=t_1$. It is known that their poles as functions of $t_2$ move as particles of the elliptic Calogero-Moser model. We extend this correspondence to the level of hierarchies and find the Hamiltonian $H_k$ of the elliptic Calogero-Moser model which governs the dynamics of poles with respect to the $k$-th hierarchical time. The Hamiltonians $H_k$ are obtained as coefficients of the expansion of the spectral curve near the marked point in which the Baker-Akhiezer function has essential singularity.