[Math] How to two dependent variables have zero covariance

conditional probabilityconditional-expectationcovarianceexpected valueprobability distributions

The following text is from page 59 of chapter 3 of the book Deep Learning by Ian Goodfellow et al.:

Independence is a stronger requirement than zero covariance, because independence also excludes nonlinear relationships. It is possible for two variables to be dependent but have zero covariance.

For example, suppose we first sample a real number $x$ from a uniform distribution over the interval [−1,1]. We next sample a random variable $s$. With probability $\frac{1}{2}$, we choose the value of $s$ to be 1. Otherwise, we choose the value of $s$ to be −1. We can then generate a random variable $y$ by assigning $y=sx$. Clearly, $x$ and $y$ are not independent, because $x$ completely determines the magnitude of $y$. However, $Cov(x, y) = 0$.

What is the difference between sampling a real number and sampling a random variable?
How can I mathematically show that $Cov(x,y)=0$?

Best Answer

It means the same thing. It's just talking about sampling a number from the uniform distribution, call it X, on [-1,1], i.e. from $X\sim Unif[-1,1]$.

To calculate the covariance, you might note the formula $Cov(X,Y)=E(XY)-E(X)E(Y)$. It is known that E(X)=0, beacause X is uniform on [-1,1], so pick the middle value. $E(Y)=E(SX)=E(S)E(X)$ (beacuse they are independent), and their individual expectations are 0, so E(Y)=0. We need to find E(XY) now. Note that $E(XY)=E(SX^2)=E(S)E(X^2)$ again because S and X are independent. So we get $0\cdot E(X^2)=0$. Thus $Cov(X,Y)=0$.

To show that X and Y are dependent events, note that the value of Y is half-determined by the value of X. So Y "depends" on X. To show mathematically that they are dependent, you must find sets A and B such that $X\in A$ and $Y\in B$ are events, where $P(Y\in B|X\in A)\ne P(Y\in B)$. Choose $A=[.25,.75],B=[.25,.75]$. Then $P(Y\in B)=\frac 1 4$ (this is because Y is also uniformly distributed on [-1,1]). But $P(Y\in B|X\in A)=\frac 1 2$. This is because given that $X\in[.25,.75]$, there is equal chance that $Y\in[.25,.75]$ or $Y\in[-.75,-.25]$. Thus we have found an events such that $P(Y\in B|X\in A)\ne P(Y\in B)$, and the random variables are dependent.

Two variables are independent if $P(X\in A\cap Y\in B)=P(X\in A)P(Y\in B)$ for all possible combinations of sets A, B such that $X\in A$ and $Y\in B$. Equivalently, that $P(X\in A|Y\in B)=P(X\in A)$ for all possible combinations of events. This is obtained by dividing the first equation by $P(Y\in B)$.