The center of our Galaxy is known to host a massive compact object, Sgr A$^*$, which is commonly considered as a super-massive black hole of $sim 4times 10^6$ M$_odot$. It is surrounded by a dense and massive nuclear star cluster, with a half mass radius about $5$~pc and a mass larger than $10^{7}$ M$_odot$. In this paper we studied the evolutionary fate of a very dense cluster of intermediate mass black holes, possible remnants of the dissipative orbital evolution of massive globular cluster hosts. We performed a set of high precision $N$-body simulations taking into account deviations from pure Newtonian gravitational interaction via a Post Newtonian development up to $2.5$ order, which is the one accounting for energy release by gravitational wave emission. The violent dynamics of the system leads to various successive merger events such to grow a single object containing $sim 25$ per cent of the total cluster mass before partial dispersal of the cluster, and such to generate, in different bursts, a significant quantity of gravitational waves emission. If generalized, the present results suggest a mechanism of mass growth up to the scale of a super massive black hole.