I am trying to derive the expectation $\mathbb E$ of the sample covariance
$$\overline{cov}_{X,Y} := \frac{1}{n-1}\cdot \sum_{i=1}^n (X_i-\overline X)(Y_i – \overline Y)$$
where $\overline X = \frac1n \sum_{i=1}^n X_i$ and $\overline Y = \frac1n \sum_{i=1}^n Y_i$ and $X_1,\ldots, X_n$ are some iid random variables, and so are $Y_1,\ldots,Y_n$.
I start as follows:
$\mathbb E[\overline{cov}_{X,Y}] = \frac{1}{n-1}\cdot \mathbb E[\sum_{i=1}^n(X_i Y_i – X_i\overline Y – Y_i\overline X + \overline X\overline Y)]$
$\quad\quad\quad\quad = \frac{1}{n-1}\cdot (n\cdot \mathbb E[X_1 Y_1] – n\mathbb E[\overline X\overline Y] – n\mathbb E[\overline Y\overline X] + n\mathbb E[\overline X\overline Y])$
$\quad\quad\quad\quad = \frac{n}{n-1}\cdot (\mathbb E[X_1 Y_1] – \mathbb E[\overline X\overline Y])$
since $n\overline X = \sum_{i=1}^n X_i$ and $n\overline Y = \sum_{i=1}^n Y_i$.
Then I check
$E[\overline X\overline Y] = \frac{1}{n^2}\cdot \mathbb E[\sum_{i=1}^n\sum_{j=1}^nX_i Y_j]$
$\quad\quad\quad = \frac{1}{n^2}\cdot \sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_i Y_j] \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad\ (1)$
$\quad\quad\quad = \frac{1}{n^2}\cdot \sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_1 Y_1]\quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad \quad (2)$
$\quad\quad\quad = \mathbb E [X_1 Y_1]$
But this gives $\mathbb E[\overline{cov}_{X,Y}] = 0$, which can't be right. I feel like I did a mistake by going from line (1) to (2), but I cannot figure out why. I mean, e.g. we should have $\mathbb E[X_1 Y_j] = \mathbb E[X_1 Y_1]$ because all $Y_j$ are iid.
However, ideally I'd want
$$\frac{1}{n^2}\sum_{i=1}^n\sum_{j=1}^n \mathbb E [X_i Y_j] = \frac{1}{n^2}\sum_{i=1}^n\sum_{j=1, i\ne j}^n \mathbb E [X_i]\mathbb E[Y_j] +\frac{1}{n^2}\sum_{i=1}^n \mathbb E [X_i Y_i]$$
but how?
Best Answer
What exactly are the dependence relations between the $X_i$ and the $Y_j$?
If $Y_1$ and $Y_2$ are iid, then you do not necessarily have that $X_1Y_1$ and $X_1Y_2$ have the same distribution. For instance, if $X_1$ is independent of $Y_2$, but dependent on $Y_1$ (which happens e.g. if $X_1=Y_1$, in which case $X_1Y_1=Y_1^2$), the joint distributions of $(X_1,Y_1)$ and $(X_1,Y_2)$ are different, which makes the products $X_1Y_1$ and $X_1Y_2$ behave differently.
I believe that in your case, $X_i$ is independent of $Y_j$ for all $j\neq i$, but that $X_i$ and $Y_i$ are dependent, and that $(X_1,Y_1)$ and $(X_i,Y_i)$ have the same joint distribution for all $i$. This makes sense in regards to the covariance, because you sample your variables pairwise, but otherwise independently, i.e. you get data points $(X_1,Y_1), (X_2,Y_2), \ldots$.