Can I write $\text{Cov}[x,y+z]=\text{Cov}[x,y]+\text{Cov}[x,z]$,
where $\text{Cov}(.)$ is referring to the population covariance? $x,y$ and $z$ are random variables.
(very fundamental question… 🙂
Solved – Splitting covariance Cov[x,y+z]
covariance
Related Solutions
The first term (i): $E[\operatorname{cov}(X,Y|Z)]$
Think of $\operatorname{cov}(X,Y)$ as a function of $Z$. As you examine different values of $Z$, you will correspondingly get a value for $\operatorname{cov}(X,Y)$. The expectation simply averages these different covariances with respect to $Z$.
The second term (ii): $\operatorname{cov}([E[X|Z],E[Y|Z])$
Think of $E[X|Z]$ and $E[Y|Z]$ as functions of $Z$. As you examine different values of $Z$, you will correspondingly get a value of $E[X|Z]$ and a value of $E[Y|Z]$ realized simultaneously. Therefore, for every value of $Z$, you will get an $(X, Y)$ coordinate. This term is simply the covariance of all these coordinate points.
"Covariance" is used in many distinct senses. It can be
a property of a bivariate population,
a property of a bivariate distribution,
a property of a paired dataset, or
an estimator of (1) or (2) based on a sample.
Because any finite collection of ordered pairs $((x_1,y_1), \ldots, (x_n,y_n))$ can be considered an instance of any one of these four things--a population, a distribution, a dataset, or a sample--multiple interpretations of "covariance" are possible. They are not the same. Thus, some non-mathematical information is needed in order to determine in any case what "covariance" means.
In light of this, let's revisit three statements made in the two referenced posts:
If $u,v$ are random vectors, then $\operatorname{Cov}(u,v)$ is the matrix of elements $\operatorname{Cov}(u_i,v_j).$
This is complicated, because $(u,v)$ can be viewed in two equivalent ways. The context implies $u$ and $v$ are vectors in the same $n$-dimensional real vector space and each is written $u=(u_1,u_2,\ldots,u_n)$, etc. Thus "$(u,v)$" denotes a bivariate distribution (of vectors), as in (2) above, but it can also be considered a collection of pairs $(u_1,v_1), (u_2,v_2), \ldots, (u_n,v_n)$, giving it the structure of a paired dataset, as in (3) above. However, its elements are random variables, not numbers. Regardless, these two points of view allow us to interpret "$\operatorname{Cov}$" ambiguously: would it be
$$\operatorname{Cov}(u,v) = \frac{1}{n}\left(\sum_{i=1}^n u_i v_i\right) - \left(\frac{1}{n}\sum_{i=1}^n u_i\right)\left(\frac{1}{n}\sum_{i-1}^n v_i\right),\tag{1}$$
which (as a function of the random variables $u$ and $v$) is a random variable, or would it be the matrix
$$\left(\operatorname{Cov}(u,v)\right)_{ij} = \operatorname{Cov}(u_i,v_j) = \mathbb{E}(u_i v_j) - \mathbb{E}(u_i)\mathbb{E}(v_j),\tag{2}$$
which is an $n\times n$ matrix of numbers? Only the context in which such an ambiguous expression appears can tell us which is meant, but the latter may be more common than the former.
If $u,v$ are not random vectors, then $\operatorname{Cov}(u,v)$ is the scalar $\Sigma u_i v_i$.
Maybe. This assertion understands $u$ and $v$ in the sense of a population or dataset and assumes the averages of the $u_i$ and $v_i$ in that dataset are both zero. More generally, for such a dataset, their covariance would be given by formula $(1)$ above.
Another nuance is that in many circumstances $(u,v)$ represent a sample of a bivariate population or distribution. That is, they are considered not as an ordered pair of vectors but as a dataset $(u_1,v_1), (u_2,v_2), \ldots, (u_n,v_n)$ wherein each $(u_i,v_i)$ is an independent realization of a common random variable $(U,V)$. Then, it is likely that "covariance" would refer to an estimate of $\operatorname{Cov}(U,V)$, such as
$$\operatorname{Cov}(u,v) = \frac{1}{n-1}\left(\sum_{i=1}^n u_i v_i - \frac{1}{n}\left(\sum_{i=1}^n u_i\right)\left(\sum_{i-1}^n v_i\right)\right).$$
This is the fourth sense of "covariance."
If two vectors are not random, then their covariance is zero.
This is an unusual interpretation. It must be thinking of "covariance" in the sense of formula $(2)$ above,
$$\left(\operatorname{Cov}(u,v)\right)_{ij} = \operatorname{Cov}(u_i,v_j) = 0$$
Each $u_i$ and $v_j$ is considered, in effect, a random variable that happens to be a constant.
In a regression context (where vectors, numbers, and random variables all occur together) some of these distinctions are further elaborated in the thread on variance and covariance in the context of deterministic values.
Best Answer
Yes, covariance is linear in either of its arguments, which is to say:
$\text{Cov}(aX,Y) = \text{Cov}(X,aY) = a\text{Cov}(X,Y)$ (in the univariate case; an analogous formula holds for the multivariate case) and
$\text{Cov}(A+B,C) = \text{Cov}(A,C) + \text{Cov}(B,C)\,$.
This linearity follows from the definition of covariance and the basic properties of expectation - in particular, linearity of expectation.
Let $\mu_X = E(X)$ and similarly for the other variables:
\begin{eqnarray} \text{Cov}(X,Y+Z) &=& \text{E}[(X-\text{E}(X))(Y+Z-\text{E}(Y+Z)]\\ &=& \text{E}[(X-\mu_X)(Y+Z-\{\mu_Y+\mu_Z\})]\\ &=& \text{E}[(X-\mu_X)(Y-\mu_Y+Z-\mu_Z)]\\ &=& \text{E}[(X-\mu_X)(Y-\mu_Y)]+\text{E}[(X-\mu_X)(Z-\mu_Z)]\\ &=& \text{Cov}(X,Y)+\text{Cov}(X,Z) \end{eqnarray}
(which is pretty similar what you have in your own hint in comments there)