Probability Theory – Why Variance of Discrete Distribution Exceeds Variance of Discrete Uniform Distribution

discrete-distributionsdistributionsprobabilityrandom variable

I am not a mathematician, so I don't quite understand how comes that a variance of some discrete probability distribution could exceed the variance of the discrete uniform distribution. I thought that variance of any distribution will be capped by the variance of the flat distribution.

My case:
I have the following probability distribution for the discrete random variable on the [0,4] interval:
0.66$\hspace{0.5cm}$0.16$\hspace{0.5cm}$0.01$\hspace{0.5cm}$0.01$\hspace{0.5cm}$0.16

I compute the variance of this distribution using:
$$
\text{Var}(X) = \sum_{i=1}^n p_i\,\cdot\,(x_i – \mu)^2
$$
and I get 2.1275, while the variance for the discrete uniform distribution, according to this formula
$$
\text{Var}(X) = \frac{n^2 – 1}{12}, \quad \text{where }n=b-a+1
$$
should be 2.0 on the same interval, i.e., for $b-a = 4$.

This bugs me a lot. Please tell me what I am missing. Thank you for your time.

Best Answer

Popoviciu's inequality states that the sharp upper bound on the variance $\sigma^2$ of any bounded probability distribution with support on $[a, b]$ is: $$ \sigma^2\leq \frac{1}{4}(b - a)^2 $$ In your case, $a= 0, b = 4$ so we have $\sigma^2\leq 4$. This is achieved precisely when half of the probability is concentrated at each of the two bounds.

Related Solutions

Solved – How to generate numbers according to a Soliton distribution

If we start at $k=2$, the sums telescope, giving $1-1/k$ for the (modified) CDF. Inverting this, and taking care of the special case $k=1$, gives the following algorithm (coded in R, I'm afraid, but you can take it as pseudocode for a Python implementation):

rsoliton <- function(n.values, n=2) {
  x <- runif(n.values)         # Uniform values in [0,1)
  i <- ceiling(1/x)            # Modified soliton distribution
  i[i > n] <- 1                # Convert extreme values to 1
  i
}

As an example of its use (and a test), let's draw $10^5$ values for $N=10$:

n.trials <- 10^5
i <- rsoliton(n.trials, n=10)
freq <- table(i) / n.trials  # Tabulate frequencies
plot(freq, type="h", lwd=6)

Frequency distribution

Beta-Binomial Distribution – Marginal Distribution in Beta-Binomial Model with Overdispersion Parameters

Using well known formulas for the mean and variance of a beta distributed random variable, in terms of $\pi_i$ and $\tau_i^2$, $$ E(P_i)=\frac{a_i}{a_i+b_i}=\pi_i $$ and $$ \operatorname{Var}P_i=\frac{a_ib_i}{(a_i+b_i)^2(a_i+b_i+1)}=\pi_i(1-\pi_i)\tau_i^2. $$ Since $Y_i|P_i\sim \operatorname{bin}(N_i,P_i)$, using the law of total expectation, $$ E(Y_i) = E E(Y_i|P_i) = EN_i P_i=N_i \pi_i $$ and, using the law of total variance, \begin{align} \operatorname{Var}Y_i&=\operatorname{Var}E(Y_i|P_i)+E\operatorname{Var}(Y_i|P_i) \\&=\operatorname{Var}N_i P_i+E(N_i P_i(1-P_i)) \\&=N_i^2\pi_i(1-\pi_i)\tau_i^2+N_i[EP_i-(EP_i)^2-\operatorname{Var}P_i] \\&=N_i^2\pi_i(1-\pi_i)\tau_i^2+N_i[\pi_i(1-\pi_i)-\pi_i(1-\pi_i)\tau_i^2] \\&=(\tau_i^2 N_i + 1-\tau_i^2)N_i\pi_i(1-\pi_i). \end{align} As sanity checks, note that as $\tau_i^2$ tends to zero $\operatorname{Var} Y_i$ tends to the binomial variance. Similarly, as $a_i$ and $b_i$ tends to zero, $\tau_i^2$ tends to one and $\operatorname{Var}Y_i$, as expected, tends to $N_i^2\pi_i(1-\pi_i)$, the variance of $N_i$ times a Bernoulli random variable.

Best Answer

Related Solutions

Solved – How to generate numbers according to a Soliton distribution

Beta-Binomial Distribution – Marginal Distribution in Beta-Binomial Model with Overdispersion Parameters

Related Question