Solved – Gaussian covariance matrix basic concept

linear algebranormal distribution

We have a $$ \boldsymbol\mu= \begin{bmatrix} \mu_1 \\ \mu_2 \end{bmatrix} $$

$$ \boldsymbol Y= \begin{bmatrix} Y_1 \\ Y_2 \end{bmatrix} $$

with

$$
\begin{bmatrix}
\Sigma_{11} & \Sigma_{12}\\
\Sigma_{21} & \Sigma_{22}
\end{bmatrix}
$$

Why they represent covariance with 4 separated matrix?

and what does $$ \Sigma_{11} \Sigma_{12} \Sigma_{21} \Sigma_{22} $$
seperately mean in Gaussian?

Best Answer

Why they represent covariance with 4 separated matrices? I emphasize this each notion as matrix. what happen if each notion become a matrix

In this case the vectors ${\boldsymbol Y}$ and ${\boldsymbol \mu}$ are really block vectors. In the case of an $n$-dimensional ${\boldsymbol Y}$ vector we could expand it as follows:

$$\boldsymbol Y= \begin{bmatrix} \color{blue}{Y_1} \\ \color{red}{Y_2} \end{bmatrix}=\begin{bmatrix}\color{blue}{Y_{11}\\Y_{12}\\\vdots\\ Y_{1h}}\\\color{red}{Y_{21}\\Y_{22}\\\vdots\\ Y_{2k}}\end{bmatrix}\tag{$n \times 1$}$$

showing the partition of the $n$ coordinates into two groups of size $h$ and $k$, respectively, such that $n = h + k$. A parallel illustration would immediately follow for the $\boldsymbol \mu$ vector of population means.

The block matrix of covariances would hence follow as:

$$\begin{bmatrix} \Sigma_{\color{blue}{11}} & \Sigma_{\color{blue}{1}\color{red}{2}}\\ \Sigma_{\color{red}{2}\color{blue}{1}} & \Sigma_{\color{red}{22}} \end{bmatrix} \tag {$n \times n$}$$

where

$$\small\Sigma_{\color{blue}{11}}=\begin{bmatrix} \sigma^2({\color{blue}{Y_{11}}}) & \text{cov}(\color{blue}{Y_{11},Y_{12}}) & \dots & \text{cov}(\color{blue}{Y_{11},Y_{1h}}) \\ \text{cov}(\color{blue}{Y_{12},Y_{11}}) & \sigma^2({\color{blue}{Y_{12}}}) & \dots & \text{cov}(\color{blue}{Y_{12},Y_{1h}}) \\ \vdots & \vdots & & \vdots \\ \text{cov}(\color{blue}{Y_{1h},Y_{11}}) & \text{cov}(\color{blue}{Y_{1h},Y_{12}}) &\dots& \sigma^2({\color{blue}{Y_{1h}}}) \end{bmatrix} \tag{$h \times h$}$$

with

$$\small \Sigma_{\color{blue}{1}\color{red}{2}}= \begin{bmatrix} \text{cov}({\color{blue}{Y_{11}}},\color{red}{Y_{21}}) & \text{cov}(\color{blue}{Y_{11}},\color{red}{Y_{22}}) & \dots & \text{cov}(\color{blue}{Y_{11}},\color{red}{Y_{2k}}) \\ \text{cov}({\color{blue}{Y_{12}}},\color{red}{Y_{21}}) & \text{cov}(\color{blue}{Y_{12}},\color{red}{Y_{22}}) & \dots & \text{cov}(\color{blue}{Y_{12}},\color{red}{Y_{2k}}) \\ \vdots & \vdots & & \vdots \\ \text{cov}({\color{blue}{Y_{1h}}},\color{red}{Y_{21}}) & \text{cov}(\color{blue}{Y_{1h}},\color{red}{Y_{22}}) & \dots & \text{cov}(\color{blue}{Y_{1h}},\color{red}{Y_{2k}}) \end{bmatrix}\tag{$h \times k$} $$

its transpose...

$$\small \Sigma_{\color{red}{2}\color{blue}{1}}= \begin{bmatrix} \text{cov}({\color{red}{Y_{21}}},\color{blue}{Y_{11}}) & \text{cov}(\color{red}{Y_{21}},\color{blue}{Y_{12}}) & \dots & \text{cov}(\color{red}{Y_{21}},\color{blue}{Y_{1h}}) \\\text{cov}({\color{red}{Y_{22}}},\color{blue}{Y_{11}}) & \text{cov}(\color{red}{Y_{22}},\color{blue}{Y_{12}}) & \dots & \text{cov}(\color{red}{Y_{22}},\color{blue}{Y_{1h}}) \\ \vdots & \vdots & & \vdots \\ \text{cov}({\color{red}{Y_{2k}}},\color{blue}{Y_{11}}) & \text{cov}(\color{red}{Y_{2k}},\color{blue}{Y_{12}}) & \dots & \text{cov}(\color{red}{Y_{2k}},\color{blue}{Y_{1h}}) \end{bmatrix}\tag{$k \times h$} $$

and

$$\small \Sigma_{\color{red}{22}}=\begin{bmatrix} \sigma^2({\color{red}{Y_{21}}}) & \text{cov}(\color{red}{Y_{21},Y_{22}}) & \dots & \text{cov}(\color{red}{Y_{21},Y_{2k}}) \\ \text{cov}(\color{red}{Y_{22},Y_{21}}) & \sigma^2({\color{red}{Y_{22}}}) & \dots & \text{cov}(\color{red}{Y_{22},Y_{2k}}) \\ \vdots & \vdots & & \vdots \\ \text{cov}(\color{red}{Y_{2k},Y_{21}}) & \text{cov}(\color{red}{Y_{2k},Y_{22}}) &\dots& \sigma^2({\color{red}{Y_{2k}}}) \end{bmatrix} \tag{$k \times k$}$$

These partitions come into play in proving that the marginal distributions of a multivariate Gaussian are also Gaussian, as well as in the actual derivation of marginal and conditional pdf's.

Related Question