Covariance – Random Variable and Sum of Random Variables

conditional probabilitycovariancepoisson distributionprobability

Consider a vector $\{v_1,…,v_n\}$, $v_k\in B$, where $B$ consists of $B_1,…,B_m$ disjoint subsets and let
\begin{align*}
L(v_k) = \begin{cases} 1 \quad \text{, if a specific event occurs for} \quad v_k \newline
0 \quad \text{, else} \end{cases}
\end{align*}

The probability that a specific event occurs for $v_k$ is $p_i\in(0,1)$. The probabilities $p_i$, $i\in\{1,…,m\}$, are exposed to random fluctuations $\vartheta_i$, $i\in\{1,…,m\}$. These fluctuations are realizations of random variables $\theta_i$, $i\in\{1,…,m\}$, with properties
\begin{align}
\mathbb{E}[\theta_i]=1 \quad \text{and} \quad Cov(\theta_i,\theta_j)=\begin{cases} \sigma^2 \quad \text{, if} \quad i=j \newline \rho\sigma^2 \quad \text{, if} \quad i\neq j \end{cases}
\end{align}
Further, let
\begin{align*}
S_i:=\sum_{v_k\in B_i} L(v_k) \quad \text{and} \quad S:=\sum_{l=1}^{m} S_l
\end{align*}

Let us make the following two assumptions

\begin{align}
S_i | (\theta_1,…,\theta_m) \sim Poi(n_i p_i \theta_i)\quad ,n_i \in \mathbb{N}\newline
Cov(S_i,S_j | (\theta_1,…,\theta_m)) = 0 \quad \text{, for} \quad i \neq j,
\end{align}

Question 1: How can I calculate $Cov(S_i,S)$?

Question 2: Following the calculation provided below by K. A. Buhr for $Cov(S_i,S)$, I now also tried to calculate $Var(S)$ by the law of total variance:
\begin{align}
Var(S)=\mathbb{E}[Var(S|\theta^*)]+Var(\mathbb{E}[S|\theta^*])
\end{align}
The first term is then just $\mathbb{E}[\sum_{l=1}^m n_l p_l \theta_l]=\sum_{l=1}^m n_l p_l$, assuming $\mathbb{E}[\theta_i]=1$.
The seond term becomes
\begin{align}
Var[\sum_{l=1}^m n_l p_l \theta_l]
=\sum_{l=1}^m Var(n_l p_l \theta_l)+\sum_{l\neq j} Cov[n_l p_l \theta_l,n_j p_j \theta_j]
=\sigma^2(\sum_{l=1}^m n_l^2 p_l^2 + \rho \sum_{l\neq j} n_l p_l n_j p_j)
\end{align}
Therefore,
\begin{align}
Var(S)=\sum_{l=1}^m n_l p_l+\sigma^2(\sum_{l=1}^m n_l^2 p_l^2 + \rho \sum_{l\neq j} n_l p_l n_j p_j)
\end{align}
Are my calculations for $Var(S)$ correct?

Best Answer

Note that you have Poisson rates $n_i p_i \theta_i \geq 0$ but $E(\theta_i)=0$. I'll assume you actually mean $E(\theta_i)=1$ (or equivalently, rates $n_i p_i (1+\theta_i)$ with $E(\theta_i)=0$).

Anyway, you should be able to apply the "law of total covariance": $$\mathrm{Cov}(X,Y)=E[\mathrm{Cov}(X,Y\mid Z)] + \mathrm{Cov}(E[X\mid Z], E[Y\mid Z])$$

Writing $\theta_\star:=(\theta_1,\dots,\theta_n)$, by the conditional (lack of) correlation and marginal distributions of the $S_i$, you have: $$\mathrm{Cov}(S_i,S\mid\theta_\star) = \mathrm{Var}(S_i\mid\theta_\star) = n_i p_i \theta_i$$ and so $E[\mathrm{Cov}(S_i,S\mid\theta_\star)] = n_i p_i$.

You also have $E[S_i\mid\theta_\star] = n_i p_i \theta_i$ and $E[S\mid\theta_\star] = \sum_j n_j p_j \theta_j$ giving: $$\mathrm{Cov}(E[S_i\mid\theta_\star],E[S\mid\theta_\star]) = \sum_j n_i n_j p_i p_j \mathrm{Cov}(\theta_i,\theta_j) = \sigma^2 n_i p_i \left( n_i p_i + \rho\sum_{j\not=i} n_j p_j \right)$$

The law of total covariance then gives: $$\mathrm{Cov}(S_i,S) = n_i p_i + \sigma^2 n_i p_i \left(n_i p_i + \rho \sum_{j\neq i} n_j p_j\right)$$

Related Solutions

Probability – How to Find the Variance of Dependent Variables Using Covariance and Correlation

Somewhere you stopped applying $\frac{1}{n^2}$ to the right-hand side of your expressions

You should have ended up with $$\text{Var}(\bar{X}) =\frac{1}{n^2}\left(n\text{Var}(X)+2 \frac{n(n-1)}{2} \rho\text{Var}(X)\right) =\frac{1+(n-1)\rho}{n} \text{Var}({X})$$

which, as expected, is $\frac{1}{n} \text{Var}({X})$ when $\rho=0$, and is $ \text{Var}({X})$ when $\rho=1$

Variance of sample mean of correlated random variables is zero

There is nothing wrong with your work. It is just that you have an extreme (and so slightly weird) case. Note that the pairwise correlations cannot be less than $-\frac1{n-1}$ if you want the matrix to be positive semidefinite and even then there will be restrictions on the distribution of the $X_i$ to achieve this. The extreme effect is to make $\bar X=\mu$ $(=0$ here$)$ almost surely.

For example when $n=2$ this gives $\rho=-1$ and since $X_1$ and $X_2$ have identical distributions, you get $X_2=-X_1$ with probability $1$ and so $\bar X=0$, a constant with zero variance. For this to even be possible, you need the distribution of $X_1$ and $X_2$ to be symmetric about $0$.

Much the same thing happens with larger $n$. Here is a simulation using R of ten observations from a multivariate normal distribution and taking the correlation matrix as the covariance matrix when $n=6$:

library(mvtnorm)
set.seed(2022)
n <- 6
covmatrix <- matrix(rep(-1/(n-1), n^2), ncol=n) + diag(n)*n/(n-1) 
covmatrix
#      [,1] [,2] [,3] [,4] [,5] [,6]
# [1,]  1.0 -0.2 -0.2 -0.2 -0.2 -0.2
# [2,] -0.2  1.0 -0.2 -0.2 -0.2 -0.2
# [3,] -0.2 -0.2  1.0 -0.2 -0.2 -0.2
# [4,] -0.2 -0.2 -0.2  1.0 -0.2 -0.2
# [5,] -0.2 -0.2 -0.2 -0.2  1.0 -0.2
# [6,] -0.2 -0.2 -0.2 -0.2 -0.2  1.0
sims <- rmvnorm(10, mean=rep(0,n), sigma=covmatrix)
sims
#             [,1]        [,2]        [,3]        [,4]       [,5]        [,6]
# [1,]  2.05353690 -0.21785513  0.08433481 -0.51489124  0.7048736 -2.10999909
# [2,] -1.34855528  0.11628539  0.63282192  0.07644164  0.9140224 -0.39101608
# [3,] -0.59594809  0.58136474  0.42176678  0.39159438 -0.2369455 -0.56183242
# [4,]  0.21283317  0.03699741 -0.50479400 -0.48377217  0.3156341  0.42310169
# [5,] -0.20142159 -0.76144379  0.89221894  0.53952061 -0.3872322 -0.08164199
# [6,]  0.23320763  1.01507018 -1.60352276 -0.13737918 -0.7910043  1.28362840
# [7,]  0.86343020 -0.16358892 -0.14731998 -0.50410625 -0.6139985  0.56558351
# [8,] -0.64649148 -0.98177522  0.65169196 -0.28309427  0.9583281  0.30134078
# [9,] -2.00950883 -0.17680495  1.27467798  0.51076605 -0.3755420  0.77641190
#[10,] -0.04605967  0.31143296 -0.35956513  0.90532379 -0.8817965  0.07066465
Xbar <- rowMeans(sims)
Xbar
# [1] -3.246942e-08  5.724417e-09 -1.458760e-08  2.749809e-08 -5.479594e-09
# [6] -3.169053e-09  9.711573e-09 -1.781134e-08  1.764540e-08  1.170931e-08
var(Xbar)
# [1] 3.284356e-16

so neither the simulated $\bar X$s nor their variance are precisely $0$ but close enough to make the point.

Best Answer

Related Solutions

Probability – How to Find the Variance of Dependent Variables Using Covariance and Correlation

Variance of sample mean of correlated random variables is zero

Related Question