I know this sounds like something I can just Google, but surprisingly, I cannot find it ANYWHERE.
All I found is a document that says: "We will skip the variance of categorical distribution here."
Source: https://livebook.manning.com/book/math-and-architectures-of-deep-learning/chapter-5/v-5/
I know Categorical Distribution is a generalization of the Bernoulli distribution (variance: $p(1−p)$). So what's the variance of Categorical Distribution? $p_i(1-p_i)$?
Thank you
Distribution Definition: https://en.wikipedia.org/wiki/Categorical_distribution
Best Answer
The definition of the variance of a random variable $X$ is (when it exists) $$ \operatorname{Var}[X] = \mathbb{E}[(X-\mathbb{E}[X])^2] =\mathbb{E}[X^2]-\mathbb{E}[X]^2 $$ Here, this is the same: if $X$ is distributed according to a categorical distribution $p$, we have $$ \operatorname{Var}[X] = \sum_{i=1}^k \left(i - \sum_{j=1}^k j p_j\right)^2 p_i = \sum_{i=1}^k i^2 p_i - \left(\sum_{i=1}^k i p_i\right)^2 $$ There is no general closed form for this.