Derivations for the higher tensor components of the quadrupole polarizabilities are given and their values for the metastable states of the Ca$^+$, Sr$^+$ and Ba$^+$ alkaline earth-metal ions are estimated. We also give the scalar quadrupole polarizabilities of the ground and metastable states of these ions to compare our results with the previously available theoretical and experimental results. Reasonably good agreement between our calculations with the previous values of scalar quadrupole polarizabilities demonstrate their correctness. The reported scalar and tensor quadrupole polarizabilities could be very useful to estimate the uncertainties due to the gradient of the electric fields in the clock frequencies of the above alkaline earth-metal ions when accuracies of these frequency measurements attain below 10$^{-19}$ precision level.