I'm following this tutorial on Canonical Correlation Analysis, and had a question about an example from that tutorial.
Question:
On Page 4 of that document, the following example is given:
"Consider two normally distributed two-dimensional variables $\mathbf{x}$ and $\mathbf{y}$ with unit variance. Let $y_1 + y_2 = x_1
+ x_2$. It is easy to confirm that the correlation matrix between $\mathbf{x}$ and $\mathbf{y}$ is"
\begin{equation*}
R_{xy} =
\begin{pmatrix}
0.5 & 0.5 \\
0.5 & 0.5 \\
\end{pmatrix}
\end{equation*}
Attempt:
I am unclear about how this was derived. My understanding is that the entries in this matrix are $\rho_{i,j}$, ie. they are normalized by the variances of the variables.
For entry (1,1) in the matrix, I wrote out the expression:
$$
\rho_{1,1}= \frac{\mathbb{E}[x_1x_2] – \mathbb{E}[x_1]\mathbb{E}[x_2]}{\sqrt{(Var(x_1)Var(x_2)}}
$$
but I am unclear whether these variables are zero-mean (cancelling out the second term in the numerator) and how to work with $\mathbb{E}[x_1x_2]$ given the constraint on the components.
Thanks!
Best Answer
The key insights are (a) Normality is irrelevant, because the problem concerns correlations only; and therefore (b) we shouldn't have to rely on formulas for conditional correlations of Normally distributed (vector) random variables. What is left, then? Only the clear parallels (symmetries) between the roles of the $X_i$ and $Y_i.$ This leads directly to a simple solution.
To comport with conventions, I will write the random variables using capital letters.
Let's suppose $(X_1,X_2,Y_1,Y_2)$ is a vector random variable with unit covariance matrix: that is, all marginals have unit variance and otherwise are uncorrelated. (This adds the assumption of uncorrelation to the problem statement but weakens the Normality assumption.)
The question concerns a conditional covariance for the conditioning event
$$*: X_1+X_2 = Y_1+Y_2.$$
We will compute the conditional variance in two ways, both of which exploit the bilinearity of covariance.
"Taking out what is known," we may treat $Y_1+Y_2$ as the same as $X_1+X_2,$ whence $$\begin{aligned}\operatorname{Cov}(X_1+X_2, Y_1+Y_2\mid *) &= \operatorname{Cov}(X_1+X_2, X_1+X_2) \\&=\operatorname{Var}(X_1) + 2\operatorname{Cov}(X_1,X_2)+\operatorname{Var}(X_2) \\&= 1 + 2(0) + 1 = 2.\end{aligned}$$
Notice the appearance of the covariance of the $X_i:$ this is one indication of the need for the additional uncorrelation assumption.
Using only bilinarity, expand the conditional covariance above as $$\cdots = \operatorname{Cov}(X_1,Y_1\mid *) + \operatorname{Cov}(X_2,Y_1\mid *) + \operatorname{Cov}(X_1,Y_2\mid *) + \operatorname{Cov}(X_2,Y_2\mid *).$$
Note that the entire problem is symmetric upon (independently) interchanging the $X_i$ or interchanging the $Y_i,$ because such switches (a) do not change the (unconditional) covariance matrix and (b) do not change the conditioning event $*: X_1+X_2=Y_1+Y_2.$ Accordingly, all four of the terms in $(2)$ are equal. Equating $(1)$ and $(2),$ we find that four times any one of those terms equals $2.$ Solving,
$$ \operatorname{Cov}(X_i,Y_j\mid *) = \frac{2}{4} = 0.5$$
for any $i=1,2$ and $j=1,2.$
If we were to arrange this result in matrix form, with rows for $\mathbf X = (X_1,X_2)^\prime$ and columns for $\mathbf Y = (Y_1,Y_2)^\prime,$ it would read
This equals the correlation $R_{XY}$ because, even after conditioning on $*,$ the variables still have unit variance.
If we do not assume the four original variables are uncorrelated, the result is not generally correct. To appreciate this subtlety, let $Z$ be any unit-variance random variable, notice that $(X_1,X_2,Y_1,Y_2)=(Z,Z,Z,Z)$ satisfies all required conditions in the question, and because $\mathbf X = \mathbf Y$ trivially is always true, compute
$$\operatorname{Cov}(\mathbf X, \mathbf Y \mid *) = \pmatrix{1 & 1 \\ 1 & 1}.$$