Assume a data set $X$ which contains $k$ iid random vectors of size $p$. Denote by $S$ the sample covariance matrix.
Really I have some questions and I need your very appreciated opinions:
1) Ledoit and Wolf (2003) considered that the sample covariance $S$ is also can be written as $S=k^{-1} X(I – k^{-1} 11')X'$ where $1$ is a comfortable vector of ones. They demonstrate the reason of why $S$ becomes not invertible when $p$ is $>=k$. By the way, the reason was that under the case $k>p+1$, the rank of $S$ is at most equal to the rank of the matrix $I – k^{-1} 11'$ which is $k-1$. So when $p$ exceeds $k$, the rank of $S$ becomes deficient, and so $S$ becomes singular.
In fact, I didn't understand how $k^{-1} X(I – k^{-1} 11')X'$ is equal to $k^{-1} XX'$ ? And why when $p>=k$, the matrix becomes deficient?
2) To handle the problem of $p>=k$, Ledoit and Wolf(2004) highlight on the problem of creating a weighted average between the sample covariance matrix which has little or no structure and other (called shrinkage target) which has a lot of structure (also called structured matrix). They also mentioned that those estimated coefficients in $S$ thar are extremely high tend to contain a lot of positive error, and the extreme low cofficients tend to contain a lot of negative error.
I didn't understand what was meant by "has little or no structure" and by "a lot of structure" ?? Furthermore, how can we explain in another way the fact of extreme coefficients of $S$?
3) When $p>=k$, the sample covariance matrix is not positive definite (and in fact it will be semi-positive definite). In this case the eigenvalues of $S$ will spread out a lot.
I know that the some eigenvalues will erroneously be equal to zero. But what was meant by "Spread out"? can someone gives me more details and explanations about that.
Best Answer
I will answer most of the bottom line questions, but not all of your individual questions.
Covariance matrix with no structure means no a prior information or assumptions, other than it is positive semi-definite (psd), which all covariance matrices must be. Structure means some kind of a priori information or assumptions about one or more entries in the covariance matrix. For example, if it is known that the covariance matrix is diagonal, or has all diagonal elements equal, or has all off-diagonal elements equal, or perhaps a bound on absolute value of correlation coefficients, etc.
Eigenvalues being spread out, given that the subject of discussion is covariance matrices, i.e., psd, means that the 2-norm condition number, which is the ratio of the largest eigenvalue to the smallest eigenvalue, is "large". The eigenvalues will be the most spread out possible if the 2-norm condition number = infinity, which happens exactly when one or more eigenvalues = 0 (presuming there is at least one positive eigenvalue), i.e., when the covariance matrix is singular, i.e, when it is psd, but not positive definite. if there is not enough (independent) data relative to the covariance matrix dimension, and in in particular if k < p, the sample covariance matrix will be singular, i.e., have at least one eigenvalue = 0, even if the actual covariance matrix is not singular, therefore, the sample covariance matrix will have eigenvalues which are too spread out, as measured by the 2-norm condition number. The authors also claim that at least in some circumstance the largest eigenvalue in the sample covariance matrix is too large - I leave that for you to verify.
All of these schemes to adjust eigenvalues relative to the eigenvalues of the sample covariance are attempts to reduce the condition number of the unbiased sample covariance matrix estimator of covariance. There are several different ways which have been suggested in various papers over the years, beginning with Stein in 1956. they may accomplish this by introducing a bias in exchange for reducing the variance, thereby perhaps improving the Mean Square Error.