Covariance Matrix – Understanding the Asymptotic Covariance Matrix of $\bar{\pmb x}$

asymptoticscentral limit theoremcovariance-matrix

In a text I'm reading it says that we define

$$
\begin{align}
\bar{\pmb x}= \begin{bmatrix}\bar x_1 \\ \bar x_2 \end{bmatrix}
\end{align}
$$
And then immediately says the asymptotic covariance matrix for $\bar{\pmb x}$ is
$$
\begin{align}
\pmb V = N E\left[(\bar{\pmb x} – \pmb \mu)(\bar{\pmb x} – \pmb \mu)' \right]
\end{align}
$$

But I'm having a hard time seeing that. I'm comfortable with the covariance matrix for $\pmb x$ being $E[(\pmb x – \pmb \mu)(\pmb x – \pmb \mu)']$ but for some reason I'm having a hard time seeing how $N$ comes into the picture even though I know it gets there via the CLT.

I know that the CLT says $\bar x \sim N(\mu, \frac{\sigma^2}{N})$ but still having a hard time getting from there to the definition of the asymptotic covariance matrix.

Distributional Assumptions

$x_i$ is distributed multi-variate normal, independently, and i.i.d.

Solution

It makes sense now, the variance isn't for $\pmb x$ but for $\pmb{\bar x}$. Therefore we have:

$\bar x \sim N(\mu, \sigma^2_{\bar x})$ which when we demean and multiply by $\sqrt{N}$ we have $\sqrt{N}(\bar x-\mu) \sim N(0, N\sigma^2_{\bar x})$.

Best Answer

Asymptotically, $\bar{x} - \mu$ goes to zero. So the asymptotical covariance of $\bar{x}$ refers to that of $\sqrt{n}(\bar{x} - \mu)$, which is $V$, when we assume all the regularity conditions of CLT are met.

By the way, we do not really need the distributional assumption on $x$.

Edit: (Further details as requested by OP)

$V = Var(\sqrt{n}(\bar{x} - \mu)) = n Var(\bar{x}) = n E (\bar{x}-\mu)^2$ which is what you have in your textbook.

In the event that $x_t$ are i.i.d, then $n Var(\bar{x}) = Var(x)$. Maybe this is what confused you? The textbook is written down without assuming i.i.d.

Related Solutions

Solved – Definition of asymptotic variance

You are missing a lot of context here (presumably taken from linear regression) so I will focus solely on the asymptotic distribution:$^\dagger$

$$\sqrt N (\sigma^2 - \hat{\sigma}^2) \overset{\text{Dist}}{\rightarrow} \text{N}(0, 2 \sigma^4). \quad \quad$$

The left-hand-side here is a scaled version of the estimation error, which is affected by the size $N$. As $N \rightarrow \infty$ we obtain convergence in distribution to the distribution shown on the right-hand-side, which does not depend on $N$. This is sensible because although the distribution of the scaled estimation error should depend on $N$, its limit should not.

It is possible to re-frame the asymptotic result as an approximating distribution that becomes more and more accurate in the limit. (Indeed, this is the main value of an asymptotic distribution.) If $N$ is large we have:

$$\ \ \quad \sigma^2 - \hat{\sigma}^2 \overset{\text{Approx}}{\sim} \text{N} \bigg( 0, \frac{2 \sigma^4}{N} \bigg).$$

As you can see from this approximating distribution, the distribution of the estimation error is centred around zero (reflecting an unbiased estimator). As $N$ becomes larger, the (approximate) distribution of the estimation error has a lower variance, so it tends to get smaller. This gives us some useful consistency properties for the estimator, and it accords with our intuition that estimator becomes more accurate as we get more data.

$^\dagger$ Note that this notation is shorthand for the following formal mathematical statement:

$$\quad \quad \lim_{N \rightarrow \infty} \mathbb{P} \Big( \sqrt N (\sigma_p^2 - \hat \sigma^2) \leqslant \epsilon \Big) = \Phi \bigg( \frac{\epsilon}{\sqrt{2} \sigma^2} \bigg) \quad \quad \quad \text{for all } \epsilon \in \mathbb{R}.$$

where the function $\Phi$ is the cumulative distribution function for the standard normal distribution.

Solved – Calculating partitioned covariance matrices using R

You can make your life a lot easier by using R's covariance function cov. So, all you need to do is read in the data to a dataframe, change the order of the columns, and then compute the covariance matrix. The partitions can then be plucked from the main covariance matrix. Here is some code to get $S$, $Sxx$, $Syx$.

> #Read in Data
> mydf<-read.table("D:\\T3_8_SONS.DAT")
> names(mydf)<-c("y1", "y2", "x1", "x2")
> #Change column order so S is created in the same order as in OP's question
> mydf2<-data.frame(mydf$x1, mydf$x2, mydf$y1, mydf$y2)
> names(mydf2)<-c("x1", "x2", "y1", "y2")
> #Print to compare to Table 3.8 in book
> head(mydf2)
   x1  x2  y1  y2
1 179 145 191 155
2 201 152 195 149
3 185 149 181 148
4 188 149 183 153
5 171 142 176 144
6 192 152 208 157
> #Obtain full variance covariance matrix
> S <- cov(mydf2)
> #Obtain covariance partitioned matrices
> Sxx <- S[1:2,1:2]
> Syy <- S[3:4,3:4]
> Syx <- Sxy <- S[1:2, 3:4]
> S
          x1       x2       y1       y2
x1 100.80667 56.54000 69.66167 51.31167
x2  56.54000 45.02333 46.11167 35.05333
y1  69.66167 46.11167 95.29333 52.86833
y2  51.31167 35.05333 52.86833 54.36000
> Sxx
         x1       x2
x1 100.8067 56.54000
x2  56.5400 45.02333
> Syy
         y1       y2
y1 95.29333 52.86833
y2 52.86833 54.36000
> Syx
         y1       y2
x1 69.66167 51.31167
x2 46.11167 35.05333

We could use your method, which is mathematically correct, to obtain $Syx$ too, but it's more work. Here are the alternate calculations:

> #Your method
> n<-25
> xbar<-apply(mydf2, 2, mean)[1:2]
> ybar<-apply(mydf2, 2, mean)[3:4]
> Syx2<-(t(mydf2[1:2])%*%as.matrix(mydf2[3:4])-n*(xbar%*%t(ybar)))/(n-1)
> Syx2
         y1       y2
x1 69.66167 51.31167
x2 46.11167 35.05333
>

Note that the $Syx$ here matches the computations I provided previously for $Syx$.