I've been having trouble figuring out how to directly prove that the covariance of the multivariate Gaussian distribution $p(x) = \dfrac{1}{{{(2\pi)}^{\frac{n}{2}}}|\Sigma|^{\frac{1}{2}}}\exp\{{-\dfrac{1}{2}(x – \mu)^T \Sigma^{-1}(x-\mu)}\}$ where $x \in \mathbb{R}^n$ and $\Sigma$ is positive definite, is actually $\Sigma$.
That is to say, given $\Sigma$ and the above density function, I want to prove that the covariance is $\Sigma$. I've been able to prove that the mean is $\mu$ by diagonalizing the matrix $\Sigma$ and integrating, but I'm struggling to do it for the proof that the covariance is $\Sigma$. Can anyone help?
Best Answer
You need to perform a linear change of variables to do this. Let $H$ be a solution to $H^TH=\Sigma^{-1}$ (it exists because $\Sigma$ is positive definite). We will consider the random vector $y=H(x-\mu)$.
We have $y^Ty=(x-\mu)^TH^TH(x-\mu)=(x-\mu)^T\Sigma^{-1}(x-\mu)$ and $|H||\Sigma|^{1/2}=1$. So the pdf of $y$ is given by
$p(y)=\frac1{|H|}p(x)=\frac{1}{|H|}\frac{1}{(2\pi)^{n/2}|\Sigma|^{1/2}}\exp\left(-\frac12(x-\mu)^T\Sigma^{-1}(x-\mu)\right)= \frac{1}{(2\pi)^{n/2}}\exp\left(-\frac12y^Ty\right)$
This means that the entries of $y$ are independent standard normal variables, so $\mathrm{Cov}[y]=I$.
In general we have $\mathrm{Cov}[Ax+b]=A\mathrm{Cov}[X]A^T$ (this is the multivariate version of $\mathrm{Var}(ax+b)=a^2\mathrm{Var}(x)$, and can be shown straightforwardly by considering the definition $\mathrm{Cov}[X]=\mathbb{E}[XX^T]-\mathbb{E}[X]\mathbb{E}[X]^T$).
So in our case $I=\mathrm{Cov}[y]=\mathrm{Cov}[H(x-\mu)]=H\mathrm{Cov}[x]H^T$, and $\mathrm{Cov}[x]=H^{-1}(H^T)^{-1}=(H^TH)^{-1}=\Sigma$.