The quadratic Jahn-Teller effect of C$_{60}^{n-}$ ($n=$ 1-5) is investigated from the first principles. Employing the density functional theory calculations with hybrid functional, the quadratic vibronic coupling constants of C$_{60}^-$ were derived. The warping of the adiabatic potential energy surface of C$_{60}^-$ by the quadratic vibronic coupling is estimated about 2 meV, which is much smaller than the Jahn-Teller stabilization energy ($approx$ 50 meV). Because of the selection rule and the vibronic reduction, the quadratic coupling slightly modifies the vibronic states of C$_{60}$ anions. Particularly, in the case of C$_{60}^{3-}$, parity and symmetry selection rule significantly reduces the effect of quadratic coupling on vibronic states. The present results confirm that the low-energy vibronic dynamics of C$_{60}^{n-}$ is of pseudorotational type.