I am solving the next exercise, but I have spent a lot of time and I can´t.
For random vectors, $X_1,X_2,…\in\mathbb{R}^p$ The sample covariance with $\Sigma_1=0$ is given by:
$$\hat\Sigma_n=\frac{1}{n-1}\sum_{i=1}^n(X_i-\hat{\mu_n})(X_i-\hat{\mu_n})^T$$
Prove that $\hat\Sigma_n=\frac{n-1}{n-2}\hat\Sigma_{n-1}+\frac{1}{n}\Delta_n\Delta_n^T$
Where $\Delta_n=X_n-\hat\mu_{n-1}$
I have the next part:
I start with the right side of the equality that I need to prove. i.e:
$$=\frac{n-2}{n-1}(\frac{1}{n-2}\sum_{i=1}^{n-1}(X_i-\hat{\mu}_{n-1})(X_i-\hat{\mu}_{n-1})^T)+\frac{1}{n}\Delta_n\Delta_n^T+\frac{1}{n}\Delta_n\Delta_n^T\\
=\frac{1}{n-1}(\sum_{i=1}^{n-1}X_iX_i^T-\sum_{i=1}^{n-1}X_i\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\sum_{i=1}^{n-1}X_i^T+\sum_{i=1}^{n-1}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T)+\frac{1}{n}\Delta_n\Delta_n^T$$
$$=\frac{1}{n-1}\sum_{i=1}^{n-1}X_iX_i^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T-\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T+\frac{1}{n}\Delta_n\Delta_n^T$$
In the previous line I used the fact that $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$ then I conclude that $\sum_{i=1}^{n-1}X_i=(n-1)\hat\mu_{n-1}$ and with the common denominator it is simplified.
From here, I have replaced with the definition given of $\Delta_n$ and $\hat\mu_{n-1}=\frac{1}{n-1}\sum_{i=1}^{n-1}X_i$
But, I can´t conclude the exercise.
Additionally, I have a question related, At some stage of doing this proof I got something like that:
$X_n\hat\mu_{n-1}^T$ and $\hat\mu_{n-1}X_n^T$, Since I am using vectors, this is a product of vectors and I got a matrix, but both are different or equal? according to me this is different because one is the transpose of other and is not equal, I did an example in R and I didn't get the same result, I mean I got that are different, Is ok my reasoning?
Best Answer
It's easy if you write $$ \hat\Sigma_n= \frac{1}{n-1}\sum_{i=1}^nX_i X_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T. $$ Split up the sum over $n$ elements into two parts. One will involve the first $n-1$ terms (which you can make look like the previous sample covariance), and the second will involve the most recent $n$th term. Finally, find a recursive formula for $\hat{\mu}_n$, and then plug that in.
Edit: The correct formula should be $$ \hat{\Sigma}_n = \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. $$ I think you flipped that first fraction accidentally.
Here's a proof: \begin{align*} \hat{\Sigma}_n &= \frac{1}{n-1}\sum_{i=1}^n(x_i - \hat{\mu}_n)(x_i - \hat{\mu}_n)^T\\ &=\frac{1}{n-1}\sum_{i=1}^n x_ix_i^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T\\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1}\hat{\mu}_n\hat{\mu}_n^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n}{n-1} \left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1}\sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \left( \hat{\mu}_{n-1} + \frac{1}{n-1}x_n \right)\left(\frac{n-1}{n} \hat{\mu}_{n-1} + \frac{1}{n}x_n \right)^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{(n-1)n} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \frac{n-1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T + \frac{1}{n}x_n x_n^T - \frac{1}{n-1} x_n x_n^T \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T + \frac{1}{n-1}x_nx_n^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T - \frac{1}{n-1} x_n x_n^T +\left( \frac{1}{n}x_n x_n^T - \frac{1}{n}x_n\hat{\mu}_{n-1}^T - \frac{1}{n} \hat{\mu}_{n-1} x_n^T + \frac{1}{n}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T \right) \\ &= \frac{1}{n-1} \sum_{i=1}^{n-1} x_ix_i^T - \hat{\mu}_{n-1}\hat{\mu}_{n-1}^T + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\left(\frac{1}{n-2} \sum_{i=1}^{n-1} x_ix_i^T - \frac{n-1}{n-2}\hat{\mu}_{n-1}\hat{\mu}_{n-1}^T\right) + \frac{1}{n}\Delta_n\Delta_n^T \\ &= \frac{n-2}{n-1}\hat{\Sigma}_{n-1} + \frac{1}{n}\Delta_n \Delta_n^T. \end{align*}