I have a set of independent but not identically distributed dice $D_0, \dots, D_n$, with mean values $\mu_i$ and variances $\sigma_i$. Now i decide to roll one dice at random with probability $p(D_i)$, lets call this one roll $D$. I know the expected value $\mathbb{E}[D] = \sum_{i=0}^n p(D_i)\mu_i = \mu$.
I would like to know the overall variance $\mathbb{E}[(D-\mu)^2] = \sigma$ and how it is related to the 'high level variance' regarding the mean values of the dice $\sum_{i=0}^n(\mu_i-\mu)^2 p(D_i) = \sigma'$ and the individual variances $\sigma_i$. My intuition for a solution would be something like $\sigma = \sigma' + \sum_{i=0}^n \sigma_i p(D_i)$, so the overall variance is the high level variance plus the average low level variance, but I am very unsure how to model this problem to calculate a proper solution.
Best Answer
Your notation is a little confusing, let me call $p_i$ the probability of choosing die $i$. Then, calling $X$ the chosen dice, and $D$ the result of the chosen dice, we get
$$E[D]= E[E[D|X]] = \sum p_i \mu_i = \mu \tag1$$
Similarly,
$$ E[D^2] =E[E[D^2|X]] = \sum p_i E[D_i^2] = \sum p_i (\mu_i^2 + \sigma_i^2) \tag2$$
and
$$ \sigma^2=Var[D] = E[D^2]- \mu^2 = \sum p_i E[D_i^2] = \sum p_i (\mu_i^2 + \sigma_i^2)-\mu^2 \tag 3$$
Now, letting
$$s' = \sum p_i (\mu_i-\mu)^2=\sum p_i \mu_i^2-\mu^2 \tag 4$$
we get
$$\sigma^2 = \sigma_W^2 + s'$$
where $\sigma_W^2 = \sum p_i \sigma_i^2$ is the weighted average of the individual variances.
This is a standard result in statistics. In the context of statistical clustering, $\sigma_W^2$ is the analog of the "within-class scatter" and $s'$ of the "between class scatter".