This paper addresses a formation tracking problem for nonlinear multi-agent systems with time-varying actuator faults, in which only a subset of agents has access to the leaders information over the directed leader-follower network with a spanning tree. Both the amplitudes and signs of control coefficients induced by actuator faults are unknown and time-varying. The aforementioned setting improves the practical relevance of the problem to be investigated, and meanwhile, it poses technical challenges to distributed controller design and asymptotic stability analysis. By introducing a distributed estimation and control framework, a novel distributed control law based on a Nussbaum gain technique is developed to achieve robust fault-tolerant formation tracking for heterogeneous nonlinear multi-agent systems with time-varying actuator faults. It can be proved that the asymptotic convergence is guaranteed. In addition, the proposed approach is applied to task-space cooperative tracking of networked manipulators irrespective of the uncertain kinematics, dynamics, and actuator faults. Numerical simulation results are presented to verify the effectiveness of the proposed designs.