Say I have a sample where $ N < p $, where $ N $ denotes the number of observations and $ p $ the number of variables. I know that the rank of the covariance matrix is then at most $ N $. I know that it is not positive definite since it is not invertible. However, I read from Wikipedia that a covariance matrix has to be at least positive semidefinite. Can I deduce positive semidefiniteness from here?
Positive definiteness of sample covariance matrix when $ N < p $
covariancecovariance-matrix
Related Solutions
I will answer most of the bottom line questions, but not all of your individual questions.
Covariance matrix with no structure means no a prior information or assumptions, other than it is positive semi-definite (psd), which all covariance matrices must be. Structure means some kind of a priori information or assumptions about one or more entries in the covariance matrix. For example, if it is known that the covariance matrix is diagonal, or has all diagonal elements equal, or has all off-diagonal elements equal, or perhaps a bound on absolute value of correlation coefficients, etc.
Eigenvalues being spread out, given that the subject of discussion is covariance matrices, i.e., psd, means that the 2-norm condition number, which is the ratio of the largest eigenvalue to the smallest eigenvalue, is "large". The eigenvalues will be the most spread out possible if the 2-norm condition number = infinity, which happens exactly when one or more eigenvalues = 0 (presuming there is at least one positive eigenvalue), i.e., when the covariance matrix is singular, i.e, when it is psd, but not positive definite. if there is not enough (independent) data relative to the covariance matrix dimension, and in in particular if k < p, the sample covariance matrix will be singular, i.e., have at least one eigenvalue = 0, even if the actual covariance matrix is not singular, therefore, the sample covariance matrix will have eigenvalues which are too spread out, as measured by the 2-norm condition number. The authors also claim that at least in some circumstance the largest eigenvalue in the sample covariance matrix is too large - I leave that for you to verify.
All of these schemes to adjust eigenvalues relative to the eigenvalues of the sample covariance are attempts to reduce the condition number of the unbiased sample covariance matrix estimator of covariance. There are several different ways which have been suggested in various papers over the years, beginning with Stein in 1956. they may accomplish this by introducing a bias in exchange for reducing the variance, thereby perhaps improving the Mean Square Error.
To expand on Brian Borchers' answer:
Per sections 3.2 and 6.5 of "Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices", by Olivier Ledoit and Michael Wolf in The Annals of Statistics 2012, inverting (adjusted to be better conditioned) estimate of covariance may not yield the best estimate of its inverse, referred to as the Precision matrix, and vice versa. So, the inverse of the well-conditioned covariance matrix will be well-conditioned, but might not be the "optimal" estimate.
If X were the MLE (maximum likelihood estimate) of covariance matrix C, then by functional invariance of maximum likelihood estimation https://en.wikipedia.org/wiki/Maximum_likelihood , inv(X) would be the MLE of inv(C). In this case, X = sample covariance matrix, so inv(sample covariance matrix) is the MLE of inv(C). However, the adjustments to the MLE in order to get a better conditioned solution mean that the adjusted estimate is not the MLE of C, and therefore there is no a priori reason to believe that the inverse of the adjusted estimate of C will be the "best" estimate of inv(C). And that is the point of those sections in the referenced paper, at least with regard to the proposed estimator of C and what that means for the inverse of C.
Best Answer
Yes, the sample covariance matrix will still be positive semi-definite.
To see this, note that if $X\in \mathbb{R}^{N\times p}$ is the data matrix (with observations in the rows and variables in the columns), then the sample covariance matrix is $C := \frac{1}{N-1}Y^TY\in \mathbb{R}^{p\times p}$, where $Y\in \mathbb{R}^{N\times p}$ is the matrix $X$ with each column's mean subtracted from that column's entries.
Note that $C$ is symmetric as $C^T = \left( \frac{1}{N-1}Y^TY\right)^T = \frac{1}{N-1}Y^TY=C$.
Also, for any $\mathbf{v}\in\mathbb{R}^{p}$, we have
$$\begin{align*} \mathbf{v}^T C \mathbf{v}&= \frac{1}{N-1}\mathbf{v}^TY^TY\mathbf{v}\\ &= \frac{1}{N-1} \left( Y\mathbf{v}\right)^T Y\mathbf{v}\\ &= \frac{1}{N-1}\left\| Y\mathbf{v}\right\|^{2}\\ &\ge 0. \end{align*} $$
Thus the sample covariance matrix $C$ is positive semi-definite.