Binomial and negative binomial mixture, what’s the new distribution

analysisprobabilityprobability distributions

Let $X \sim \operatorname{NBin}(n,q)$ and $Y | X \sim \operatorname{Bin}(X,p)$.

I am trying to find a distribution of $Y$, so I can write: $$\mathbb{P}(Y=y)=\sum_{x=y}^\infty \binom{x}{y} p^y (1-p)^{x-y} \binom{x+n-1}{x}(1-q)^nq^x$$ but I can't get anything out of the expression.

Another approach is to look at the probability generating functions, but I don't really understand how should I progress with this one, any hint would be appreciated.

Best Answer

$$\begin{align} \Pr[Y = y] &= \sum_{x=y}^\infty \binom{x}{y} \binom{x+n-1}{x} p^y (1-p)^{x-y} (1-q)^n q^x \\ &= \binom{y+n-1}{y} \left(\frac{p}{1-p}\right)^y (1-q)^n \sum_{x=y}^\infty \binom{x+n-1}{x-y} ((1-p)q)^x \\ &= \binom{y+n-1}{y} (pq)^y (1-q)^n \sum_{x=0}^\infty \binom{x+y+n-1}{y+n-1}((1-p)q)^x \\ &= \binom{y+n-1}{y}(pq)^y (1-q)^n \frac{1}{(1-(1-p)q)^{y+n}} \\ &= \binom{y+n-1}{y} \left(\frac{pq}{1-q+pq}\right)^y \left(\frac{1-q}{1-q+pq}\right)^n. \end{align}$$ This of course is negative binomial.

Related Question