Solved – Expectation of a matrix for variance-covariance

covariance-matrixexpected valuematrixvariance

I am implementing a variance-covariance matrix myself and I'm having some trouble understanding what is meant by the expectation this equation:

$ Cov(X, X) = E[(X – E(x))(X – E(x))']$

From what I understand, the expectation of a matrix of variables is the expectation of the columns of the matrix:

$ X = \left( \begin{array}{ccc}
1 & 4 & 7 \\
2 & 5 & 8 \\
3 & 6 & 9 \\
4 & 3 & 2 \end{array} \right) $

$E(X) = \left(\begin{array}{ccc} 2.5 & 4.5 & 6.5 \end{array}\right)' $

I would expect the variance-covariance matrix to be a $3×3$ matrix but using this definition of expectation $(X – E(x))$ is a $4×3$ matrix, $(X – E(x))'$ is a $3×4$ matrix, $(X – E(x))(X – E(x))'$ is therefore a $3×3$ matrix but the the expectation of this is going to be a $3×1$ column vector using the above definition.

How is this outer expectation performed so the result is a $3×3$ matrix?

Best Answer

Expectation of a matrix of variables is not the expectation of the columns of the matrix. What may confuse you is that you treat each column as a variable and calculate it's expectation estimate like an average of it's column. In this sense you are right. However covariance matrix is about covariation between this variables. So for example cell (1,3) is covariation between the third and the first variables and so on. Since you have 3 variables it can be 9 covariations (including covariation of variable with itself that is variation) that is how you get $3x3$ matrix.

So covariance matrix is:

$$\begin{bmatrix}Cov(X_{1},X_{1})& Cov(X_{1},X_{2}) &Cov(X_{1},X_{3})\\Cov(X_{2},X_{1})& Cov(X_{2},X_{2}) &Cov(X_{2},X_{3})\\Cov(X_{3},X_{1})& Cov(X_{3},X_{2}) &Cov(X_{3},X_{3})\end{bmatrix}.$$

Finally one more thing. Expectation of random matrix (i.e. matrix which elements are random variables) is a matrix of expectations of it's cells. That is if:

$$X^{*}=\begin{bmatrix}X& Y\\Z& D\\\end{bmatrix},$$

where $X$, $Y$, $Z$ and $D$ are random varibles, then

$$E(X^{*})=\begin{bmatrix}E(X)& E(Y)\\E(Z)& E(D)\\\end{bmatrix}$$

Related Solutions

Solved – Confusion about scatter matrices

What @whuber and I were saying to you in the comments are equivalent things. @whuber pointed out that the text you were reading makes points column vectors. I stuck to your own original notation where points are row vectors (this way of presenting is more common). When points are columns, thansposed ("T", or just ' in my notation) multiplier is the right one; when they are rows, it is the left one. Instead of multiplying separate vectors, it's more convenient to multiply whole matrices. See it with your data (matrix A = your "Ck"):

****** Points are rows, variables are columns [more common] ******
A
  1  2  3
  4  5  6

Column-centered A
  -1.500000000  -1.500000000  -1.500000000
   1.500000000   1.500000000   1.500000000

A'A, the scatter matrix
   4.500000000   4.500000000   4.500000000
   4.500000000   4.500000000   4.500000000
   4.500000000   4.500000000   4.500000000

****** Points are columns, variables are rows [that's how in your book] ******    
A
  1  4
  2  5
  3  6

Row-centered A
  -1.500000000   1.500000000
  -1.500000000   1.500000000
  -1.500000000   1.500000000

AA', the scatter matrix
   4.500000000   4.500000000   4.500000000
   4.500000000   4.500000000   4.500000000
   4.500000000   4.500000000   4.500000000

Solved – Variance-Covariance Matrix of the errors of a linear regression

Your assumptions in the beginning are: $$ Cov[\epsilon_i,\epsilon_j] = \sigma^2\rho $$ For the variables that correspond to observations from the same state.

Now when you calculate variance for each of the three components in the end, you have to take that into account, i.e. $$ V[\overline{\epsilon}_A] = V[\frac{1}{n_A}\sum_{i=1}^2\epsilon_{iA}] $$ $$ = \frac{1}{n_A^2}\sum_{i=1}^2\sum_{j=1}^2Cov(\epsilon_{iA},\epsilon_{jA}) $$ $$ = \frac{1}{n_A^2}(V[\epsilon_{1A}]+V[\epsilon_{2A}]+2\cdot Cov[\epsilon_{1A},\epsilon_{2A}]) $$ $$ = \frac{1}{2^2}(\sigma^2+\sigma^2+2\cdot\sigma^2\rho) $$ $$ = \frac{1}{2}\sigma^2(1+\rho) $$ This is the formula I used. Now I have shown you how the first element in the covariance matrix is calculated, you should be able to figure out how to calculate the other diagonal elements from here.

Best Answer

Related Solutions

Solved – Confusion about scatter matrices

Solved – Variance-Covariance Matrix of the errors of a linear regression

Related Question