Extreme eigenvalues of large-dimensional spiked Fisher matrices with application

Consider two $p$-variate populations, not necessarily Gaussian, with covariance matrices $Sigma_1$ and $Sigma_2$, respectively, and let $S_1$ and $S_2$ be the sample covariances matrices from samples of the populations with degrees of freedom $T$ and $n$, respectively. When the difference $Delta$ between $Sigma_1$ and $Sigma_2$ is of small rank compared to $p,T$ and $n$, the Fisher matrix $F=S_2^{-1}S_1$ is called a {em spiked Fisher matrix}. When $p,T$ and $n$ grow to infinity proportionally, we establish a phase transition for the extreme eigenvalues of $F$: when the eigenvalues of $Delta$ ({em spikes}) are above (or under) a critical value, the associated extreme eigenvalues of the Fisher matrix will converge to some point outside the support of the global limit (LSD) of other eigenvalues; otherwise, they will converge to the edge points of the LSD. Furthermore, we derive central limit theorems for these extreme eigenvalues of the spiked Fisher matrix. The limiting distributions are found to be Gaussian if and only if the corresponding population spike eigenvalues in $Delta$ are {em simple}. Numerical examples are provided to demonstrate the finite sample performance of the results. In addition to classical applications of a Fisher matrix in high-dimensional data analysis, we propose a new method for the detection of signals allowing an arbitrary covariance structure of the noise. Simulation experiments are conducted to illustrate the performance of this detector.