We identify a gravitational-dynamical process in near-Keplerian potentials of galactic nuclei that occurs when an intermediate-mass black hole (IMBH) is migrating on an eccentric orbit through the stellar cluster towards the central supermassive black hole (SMBH). We find that, apart from conventional dynamical friction, the IMBH experiences an often much stronger systematic torque due to the secular (i.e., orbit-averaged) interactions with the clusters stars. The force which results in this torque is applied, counterintuitively, in the same direction as the IMBHs precession and we refer to its action as secular-dynamical anti-friction (SDAF). We argue that SDAF, and not the gravitational ejection of stars, is responsible for the IMBHs eccentricity increase seen in the initial stages of previous N-body simulations. Our numerical experiments, supported by qualitative arguments, demonstrate that (1) when the IMBHs precession direction is artificially reversed, the torque changes sign as well, which decreases the orbital eccentricity, (2) the rate of eccentricity growth is sensitive to the IMBH migration rate, with zero systematic eccentricity growth for an IMBH whose orbit is artificially prevented from inward migration, and (3) SDAF is the strongest when the central star cluster is rapidly rotating. This leads to eccentricity growth/decrease for the clusters rotating in the opposite/same direction relative to the IMBHs orbital motion.