ProbXiv
Explorersign in
Problem archiveProblem record

Statement

Random matrix theory was introduced into theoretical physics by Wigner in the 1950s in his study of neutron scattering resonances, but as a subject, RMT goes back to the work of statisticians at the beginning of the 20th century. Recently, advances in RMT have opened the way to the statistical analysis of data sets in cases where the number of variables is comparable to the number of samples, and both are large. One might, for example, be interested in the daily temperature in hundreds of cities around the world, over a 365 day time period. At the technical level, one considers the statistics of the singular values of (appropriately centered and scaled) p×np\times n matrices M=(Mij)M=(M_{ij}), where p∼n→∞p\sim n\to\infty. Here pp is the number of variables and nn is the sample size. More precisely, one centers the MijM_{ij}'s around their sample averages, Mij→M^ij=Mij−1n∑k=1nMik,M_{ij}\to\hat M_{ij}=M_{ij}-\frac1n\sum_{k=1}^n M_{ik}, and considers the eigenvalues l1≥⋯≥lp≥0l_1\geq\cdots\geq l_p\geq0 and associated eigenvectors w1,…,wpw_1,\dots,w_p of the p×pp\times p sample matrix S=1nM^M^TS=\frac1n \hat M \hat M^T. The lil_i's and wiw_i's are known as the principal component eigenvalues and eigenvectors, respectively. In Principal Component Analysis (PCA) significant'' dimension reduction in the data occurs if the first few principal components $l_1, l_2,...$ account for a high'' proportion of the total variance tr⁡S=∑j=1plj\operatorname{tr} S=\sum_{j=1}^p l_j. A common model for the variables MijM_{ij} is to assume that they follow a (real) pp-variate Gaussian distribution Np(μ,Σ)N_p(\mu,\Sigma) with mean μ\mu and covariance matrix Σ\Sigma. Thus the columns (M1j,M2j,…,Mpj)T(M_{1j}, M_{2j},\dots, M_{pj})^T provide nn independent samples for Np(μ,Σ)N_p(\mu,\Sigma). Using recent results from RMT, much has now been proved about the statistics of l1,l2,...l_1, l_2,... as p,n→∞p,n\to\infty, p/n→γ∈(0,∞)p/n\to\gamma\in(0,\infty), in the case Σ=I\Sigma=I. In particular, we know that in the limit, l1l_1, appropriately centered and scaled, satisfies the Tracy-Widom distribution for the largest eigenvalues of a GOE matrix. However, most interesting applications involve so-called spiked populations, a terminology introduced by Johnstone, i.e. situations where most of the eigenvalues η1,...,ηp\eta_1,...,\eta_p of Σ\Sigma are equal to a common value, say 1, but the first few eigenvalues are greater than 1. Thus η1≥η2≥⋯≥ηk>ηk+1=⋯=ηp=1\eta_1\geq\eta_2\geq\cdots\geq\eta_k>\eta_{k+1}=\cdots=\eta_p=1 for some fixed k<<pk<<p. It is a major problem in multivariate analysis to analyze the statistics of the eigenvalues l1,l2,...l_1, l_2,... as p,n→∞p,n\to\infty, p/n→γ∈(0,∞)p/n\to\gamma\in(0,\infty) for such spiked populations. There are very interesting phase transitions in the theory. For example if η1>1+γ\eta_1>1+\sqrt{\gamma}, then, as p∼n→∞p\sim n\to\infty, l1l_1 emerges from the Marchenko-Pastur continuum ((γ−1)2,(γ+1)2)\bigl((\sqrt\gamma-1)^2,(\sqrt\gamma+1)^2\bigr), where most of the ljl_j's tend to accumulate, and almost surely l1→η1⋅(1+γη1−1)>(1+γ)2 .l_1\to\eta_1 \cdot \Bigl(1+\frac{\gamma}{\eta_1-1}\Bigr)>(1+\sqrt\gamma)^2\,. In the spiked, complex case, i.e. when the columns (M1j,M2j,...,Mpj)T(M_{1j},M_{2j},...,M_{pj})^T are sampled from the complex pp-variate Gaussian distribution, much is known about the asymptotic distribution of the ljl_j's, as p,n→∞p,n\to\infty, p/n→γ∈(0,∞)p/n\to\gamma\in(0,\infty). By contrast in the real case, apart from a.s. convergence of the lil_i's, very little is known about their asymptotic distributions. In the spiked, complex case the analysis is enabled by a particular technical tool, the Harish-Chandra-Itzykson-Zuber formula: unfortunately, no analog of this formula is known in the real case. While there are relatively few applications of complex spiked populations, knowledge of the asymptotic distributions of the lil_i's for real spiked populations would have immediate applications to a wide variety of problems in signal processing, genetics and finance.

Record

Added
Links

Comments

Nothing has been published against this problem yet, and nothing has been said about it.

Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.

Sign in

Solve with an agent

Open the statement in a chat, with the problem and the ground rules already written into the prompt.

This opens a third-party site. Nothing is posted back to ProbXiv and nothing you write there is recorded here — what a model gives you is an attempt, which a person still has to check.