[Math] Is the covariance matrix of multivariate normal distribution has to be positive definite

covariancenormal distribution

Let $\Sigma $ be the covariance matrix of a multivariate normal
distribution.

Thereotically, I guess it does not have to to strictly positive definition,
i.e., $\det (\Sigma )=0$ is possible.

In this case, the density function does not exist, how can I generate random
numbers(vectors) from such multivariate normal distribution?

Best Answer

A multivariate normal distribution has the density function:

$$f(X) = \frac{1}{\sqrt{(2\pi)^n |\Sigma|}} \exp\{\frac{-1}{2}X \Sigma^{-1} X^T\}$$

and the covariance should be positive definite. So if you need multivariate normal samples you've got to generate them using a valid (meaning symmetric positive definite) covariance matrix. But then, when you generated the random vectors, the calculated sample covariance can be not positive definite. As an example, consider a covariance matrix of rank 10. If you generate less than 10 sample vectors (i.e. 1 to 9 samples), and calculate their sample covariance, the calculated covariance matrix is not full rank, and its determinant becomes 0, although the generative model has a valid positive definite covariance.

Related Solutions

[Math] Sample from multivariate normal distribution with given positive-semidefinite covariance matrix

They both seem to work quite well. They give slightly different results for the estimated covariance matrices of the generated series, but I wouldn't be surprised if it's due to rounding errors somewhere in the computations.

Below is some R code which generates samples from $N(0, \boldsymbol{\Sigma})$.

n <- 10000000

X <- cbind(rnorm(n), rnorm(n))

sigma <- t(matrix(c(0.666, -0.333, -0.333, 0.666), nrow=2))
spectral <- eigen(sigma)

X.spectral <- t(spectral$vectors %*% sqrt(diag(spectral$values)) %*% t(X))
X.cholesky <- t(t(chol(sigma)) %*% t(X))
cov(X.spectral)
cov(X.cholesky)

So with my 10,000,000 samples, the covariance matrices are

> cov(X.spectral)
           [,1]       [,2]
[1,]  0.6660626 -0.3331138
[2,] -0.3331138  0.6658130
> cov(X.cholesky)
           [,1]       [,2]
[1,]  0.6660344 -0.3328923
[2,] -0.3328923  0.6656198

[Math] Find the marginal distributions (PDFs) of a multivariate normal distribution

The easiest way to convince yourself that the statement from Wikipedia you highlighted above is true is by use of the characteristic function. As we know it is defined as follows: \begin{equation} \kappa_\vec{X}(\vec{k}) := E\left[ e^{\imath \vec{k} \cdot \vec{X}} \right] \end{equation}

Now in case $\vec{X}$ is multivariate normal with means $\vec{\mu}$ and covariances ${\bf \Sigma}$,i.e. $\vec{X} \sim N(\vec{\mu},{\bf \Sigma})$ then from https://en.wikipedia.org/wiki/Multivariate_normal_distribution we have: \begin{equation} \kappa_\vec{X}(\vec{k}) = e^{\imath \vec{\mu}^T \cdot \vec{k} - \frac{1}{2} \vec{k}^T \cdot {\bf \Sigma} \cdot \vec{k}} \end{equation}

Now to marginalize in $\vec{k}$-space is easy. All we need to do is to set the respective components, i.e. those corresponding to variables we want to marginalize over in real space, of the $\vec{k}$-vector to zero. therefore we take ${\vec{k}_1} := \left( 0,\cdots 0, k_j, 0, \cdots, 0 \right)$ where the non-zero element sits at the jth position. Then clearly we have: \begin{equation} \kappa_\vec{X}(\vec{k}_1) = e^{\imath \mu_j k_j - \frac{1}{2} k_j {\bf \Sigma}_{j,j} k_j} \end{equation} Now in order to get the marginal distribution all we need to do is to invert the Fourier transform. We have: \begin{eqnarray} \rho^{(marg)}_j(x_j) &:=& \frac{1}{2\pi} \int\limits_{-\infty}^{\infty} e^{-\imath k_j x_j} \kappa_\vec{X}(\vec{k}_1) d k_j \\ &=& \frac{1}{2\pi} \int\limits_{\mathbb R} e^{-\imath k_j (x_j-\mu_j) - \frac{1}{2} {\bf \Sigma}_{j,j} k_j^2} dk_j \\ &=&e^{-\frac{1}{2} \frac{(x_j-\mu_j)^2}{{\bf \Sigma}_{j,j}} } \cdot \frac{1}{2\pi} \cdot \int\limits_{\mathbb R} e^{-\frac{1}{2} {\bf \Sigma}_{j,j} k_j^2 }dk_j \\ &=&\frac{1}{\sqrt{2 \pi {\bf \Sigma}_{j,j}}} e^{-\frac{1}{2} \frac{(x_j-\mu_j)^2}{{\bf \Sigma}_{j,j}} } \end{eqnarray} as expected.

Best Answer

Related Solutions

[Math] Sample from multivariate normal distribution with given positive-semidefinite covariance matrix

[Math] Find the marginal distributions (PDFs) of a multivariate normal distribution

Related Question