[Math] Proving positive definiteness of a covariance matrix

covariancelinear algebrapositive definite

Suppose I have a complex (covariance) matrix $\Sigma = \mathbb{E}[XX^*]$, where $X \in \mathbb{C}^n$ is a random vector with distribution $F$. I know that $\Sigma$ is positive semi-definite, with a simple proof, but I'm not sure about extending this to proving positive definiteness. In particular, I am thinking of a counter-example, but I may be misusing the linearity of $\mathbb{E}$.

A matrix $\Sigma$ is positive definite if $z^*\Sigma z > 0$ for all nonzero $z \in \mathbb{C}^n$. We can prove positive semi-definiteness by demonstrating
\begin{align*}
z^*\mathbb{E}[XX^*]z &= \mathbb{E}[z^*XX^*z]\\
&= \mathbb{E}[z^*X(z^*X)^*]\\
&= \mathbb{E}\left[|z^*X|^2\right] \geqslant 0.
\end{align*}
Now let $z$ be a sparse random vector such that, if $X = (x_1,\ldots,x_n)^T$, then $z = ((x_1^*)^{-1}, -(x_2^*)^{-1},0,\ldots,0)^T$. Clearly, $z$ is nonzero and $z^*X = x_1x_1^{-1} – x_2x_2^{-1} = 0$, thus $\mathbb{E}\left[|z^*X|^2\right] = 0$ and the matrix cannot be positive definite.

  1. I am not sure that, if $z$ is a random vector, it can "go inside" of the $\mathbb{E}[\,\cdot\,]$ operator (thus the counter-example is wrong).

  2. I am not sure that I can create $z \sim F'$ (where $F'$ has the same probability distribution as $F$ but the vectors drawn from it are sparse and have inverted, conjugated elements, as demonstrated).

  3. If this counter-example is indeed incorrect, then I believe I can prove positive definiteness, but it relies on me knowing the specifics of $F$. Am I correct in assuming that, in general (i.e. without the details of $F$), one cannot prove that $\Sigma$ is positive definite?

And if there is something else more glaringly wrong about my counter-example, please let me know. I read offhandedly that a covariance matrix is positive definite if the variables are linearly independent, but I have been unsuccessful in finding a source for that statement (with proof). If someone could provide that, I would be grateful as well.

Best Answer

For $\Sigma = \mathbb E[X X^*]$ to fail to be positive definite, there would have to be a nonzero $z \in \mathbb C^n$ with $0 = z^* \Sigma z = \mathbb E[z^* X X^* z] = \mathbb E[\|z^* X\|^2]$. Now for a nonnegative random variable to have expected value $0$, it must be $0$ almost surely. That is, there must be some $z$ such that, with probability $1$, $X$ lies in the hyperplane $\{x: z^* x = 0\}$. That may or may not be true, depending on the distribution $F$. It's easy to construct examples where your "random" vector $X$ always lies in a particular hyperplane.

Related Question