Suppose we are given a finite discrete random variable $Y$ which picks a random variable from the set $\mathcal{Y} = \{X_1, \dots, X_n \}$, the $X_i$ are finite discrete random variables over the same set $\mathcal{X}$, according to some probability distribution $\mathbb{P}$. Let $H(X)$ denote the usual Shannon entropy.
I'm interested in calculating the average entropy of $Y$, that is
$$ \mathbb{E}[H(Y)] = \sum_{X \in \mathcal{Y}} \mathbb{P} [X] H(X). $$
Is this quantity known or has it been considered before?
Thank you very much in advance!
Best Answer
To start with, I would define your average entropy as a function of $X_1,\dots,X_n$, since $Y$ is already the mixture of the $X_i$'s. So the average entropy, as you define it, cannot be expressed as a function of the density of $Y$.
Regarding your question, I believe the most meaningful interpretation of your quantity is the following. Denote by $I$ a random variable that takes values in $\{1,2,\dots,n\}$ and is distributed according to $\mathbb{P}$ and independent from $X_1,\dots,X_n$. Then, one can express $Y = X_I$, and the conditional entropy of $Y$ given $I$ is
$$ H(Y|I) = H(X_I|I) = \sum_{i=1}^n P(I=i) H(X_I|I=i) = \sum_{i=1}^n P(I=i) H(X_i). $$ This is precisely your average entropy term. This means, the average entropy is simply the entropy of $Y$ conditioned on the pick $I$.
Having said this, this quantity has probably been looked at before, in random drawing/picking experiments.