I have troubles understanding a geometrical meaning of a covariance matrix.
Let's say we have a data set containing two points (-1,1), (-1,2) and write them in to the matrix
$$D = \begin{bmatrix}
-1 & -1\\
1 & 2
\end{bmatrix}.
$$
Then we'll choose the unit-vector $v = \left( \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right)^T$.
And now, in this article is the following statement:
So, if we would like to represent the covariance matrix with a vector
and its magnitude, we should simply try to find the vector that points
into the direction of the largest spread of the data, and whose
magnitude equals the spread (variance) in this direction.If we define this vector as $v$, then the projection of our data $D$
onto this vector is obtained as $v^TD$, and the
variance of the projected data is $v^T\Sigma v$.
The covariance matrix $\Sigma$ looks like this:
$$
\Sigma = \begin{bmatrix}
0 & 0\\
0 & \frac{1}{2}
\end{bmatrix}
$$
Also, the projection of $D$ onto $v$ is the following vector:
$$
v^TD = \begin{bmatrix}
0\\
\sqrt{2} – \frac{1}{\sqrt{2}}
\end{bmatrix}
$$
So, in our case
$$
v^T\Sigma v = \left[ \frac{1}{\sqrt{2}}, \frac{1}{\sqrt{2}} \right] \begin{bmatrix}
0 & 0\\
0 & \frac{1}{2}
\end{bmatrix}
\begin{bmatrix}
\frac{1}{\sqrt{2}}\\
\frac{1}{\sqrt{2}}
\end{bmatrix} = \frac{1}{4}.
$$
So, could you, please, explain, how can $v^T\Sigma v$ be a variance of values projected on $v$? I would really appreciate "the proof", that $v^T\Sigma v$ equals variance, as I don't see, why it is true.
Best Answer
Your original data matrix containing the points (in this case, 2-dimension) is
$$ D = \begin{bmatrix} -1 & -1\\ 1 & 2 \end{bmatrix} $$
A shift of origin can be performed to center the points (-1,1) and (-1,2).
The mean of the x coordinates = -1.
The mean of the y coordinates = 3/2.
This will transform the data matrix to
$$ D = \begin{bmatrix} -1-(-1) & -1-(-1)\\ 1-\frac{3}{2} & 2-\frac{3}{2} \end{bmatrix} = \begin{bmatrix} 0 & 0\\ -\frac{1}{2} & \frac{1}{2} \end{bmatrix}$$
The covariance matrix is given by $DD^T$
$$ DD^T = \Sigma = \begin{bmatrix} 0 & 0\\ 0 & \frac{1}{2} \end{bmatrix} $$
The projection (in this case, into 1-dimension) of the points in $D$ on a vector $v$ is given by $v^TD$.
$v^TD$ can now be thought of as your 'Projected Data Matrix (P)' whose components give the coordinates of the points in the projected space (in this case, along the vector $v$).
Noting that $DD^T=\Sigma$, the covariance (variance, if 1-dimension) of $P$ is thus given by $PP^T$,
$$PP^T=v^TD(v^TD)^T=v^TDD^Tv=v^T\Sigma v$$