Solved – Sample covariance matrix and its inverse

covariancecovariance-matrixestimationlinear algebramatrix inverse

Assume we have the sample covariance matrix $S_1 = XX'/k$ which is not positive definite (in fact it is positive semi-definite) and not well conditioned in very large dimension (large $p$, small $k$).

If someone applies a specific method to the inverse of $S_1$ (that is, the inverse of the sample covariance matrix: $S_1^{-1}$) and get an inverse but which is now positive definite and well conditioned matrix. If we re-inverse the resulting matrix (the estimated inverse), do we get a positive definite and well conditioned matrix? In other words, the new sample covariance matrix is now guaranteed to be + definite and well conditioned?

Best Answer

To expand on Brian Borchers' answer:

Per sections 3.2 and 6.5 of "Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices", by Olivier Ledoit and Michael Wolf in The Annals of Statistics 2012, inverting (adjusted to be better conditioned) estimate of covariance may not yield the best estimate of its inverse, referred to as the Precision matrix, and vice versa. So, the inverse of the well-conditioned covariance matrix will be well-conditioned, but might not be the "optimal" estimate.

If X were the MLE (maximum likelihood estimate) of covariance matrix C, then by functional invariance of maximum likelihood estimation https://en.wikipedia.org/wiki/Maximum_likelihood , inv(X) would be the MLE of inv(C). In this case, X = sample covariance matrix, so inv(sample covariance matrix) is the MLE of inv(C). However, the adjustments to the MLE in order to get a better conditioned solution mean that the adjusted estimate is not the MLE of C, and therefore there is no a priori reason to believe that the inverse of the adjusted estimate of C will be the "best" estimate of inv(C). And that is the point of those sections in the referenced paper, at least with regard to the proposed estimator of C and what that means for the inverse of C.

Related Question