This article is dedicated to the following class of problems. Start with an $Ntimes N$ Hermitian matrix randomly picked from a matrix ensemble - the reference matrix. Applying a rank-$t$ perturbation to it, with $t$ taking the values $1le t le N$, we study the difference between the spectra of the perturbed and the reference matrices as a function of $t$ and its dependence on the underlying universality class of the random matrix ensemble. We consider both, the weaker kind of perturbation which either permutes or randomizes $t$ diagonal elements and a stronger perturbation randomizing successively $t$ rows and columns. In the first case we derive universal expressions in the scaled parameter $tau=t/N$ for the expectation of the variance of the spectral shift functions, choosing as random-matrix ensembles Dysons three Gaussian ensembles. In the second case we find an additional dependence on the matrix size $N$.