For any $0<p<1$ and $r$ a positive integer, the probability function
$$f(x)={{r+x-1}\choose{x}}p^r(1-p)^x \ \ \ \ \ \ x=0,1,2…$$
defines a random variable $X$.
Compute the mgf of $X$ to show that
$$m_X(u)=\Big(\frac{p}{1-(1-p)e^u)}\Big)^r \ \ \ \ \ \ \ u<\text{ln}((1-p)^{-1})$$
I have seen many solutions online, but I am still a bit unsure of how to proceed. So far I have,
$$m_X(u)=\sum_{x=1}^{\infty} e^{ux}fx$$
$$m_X(u)=\sum_{x=1}^{\infty} e^{ux}{{r+x-1}\choose{x}}p^r(1-p)^x$$
I am unsure of how to proceed. I have tried to simplify the above expression to
$${{r+x-1}\choose{x}}=\frac{(r+y-1)!}{(r-1)! \ y!}$$
But this did not yield anything promising.
I saw a solution on this site which used the identity
$${{-r}\choose{y}}=(-1)^y\frac{(r+y-1)!}{(r-1)! \ y!}$$
But I don't understand where this result has come from, nor how to prove it.
Best Answer
Let's do it your way, and then let's do it another way that may or may not be preferable. Using your notation,
$$\begin{align*} m_X(u) &= \sum_{x=0}^\infty e^{ux} \binom{r+x-1}{x} p^r (1-p)^x \\ &= \sum_{x=0}^\infty \binom{r+x-1}{x} p^r ((1-p)e^u)^x \\ &= \frac{p^r}{(1-(1-p)e^u)^r} \sum_{x=0}^\infty \binom{r+x-1}{x} (1 - (1-p)e^u)^r ((1-p)e^u)^x \\ &= \left(\frac{p}{1-(1-p)e^u}\right)^r. \end{align*}$$
Note that the lower index of summation should begin at $x = 0$ since the support of $X$ is $\{0, 1, 2, \ldots\}$. In the third step, I have pulled out a factor of $p^r$, and inserted a factor of $(1 - (1-p)e^u)^r$, neither of which depends on the variable of summation $x$. Why did I choose this factor to insert? The reason is that if we recall the PMF of a negative binomial distribution, $$\Pr[X = x] = \binom{r+x-1}{x} p^r (1-p)^x,$$ the relationship between the factors $p^r$ and $(1-p)^x$ are such that the bases must add to $1$. So long as $$0 < (1-p)e^u < 1,$$ we can think of this as a Bernoulli probability of a single trial; i.e., let $1-p^* = (1-p)e^u$, where $p^*$ is some "modified" Bernoulli probability of some other negative binomial random variable. In this way, we obtain the sum of probabilities of this "new" negative binomial random variable with parameters $p^*$ and $r$, and the sum of its probabilities over its support is also $1$.
What is the other method? Well, recall that a negative binomial random variable is simply the sum of $r$ independent and identically distributed geometric random variables; i.e., $$X = Y_1 + Y_2 + \cdots + Y_r,$$ where $Y \sim \operatorname{Geometric}(p)$, with PMF $$\Pr[Y = y] = p(1-p)^y, \quad y = 0, 1, 2, \ldots.$$ Also recall that the MGF of the sum of $r$ iid random variables is simply the MGF of one such random variable raised to the $r^{\rm th}$ power; i.e., $$m_X(u) = \left(m_Y(u)\right)^r.$$ Now if you already know that the MGF of the geometric distribution is $$m_Y(u) = \frac{p}{1-(1-p)e^u},$$ the result immediately follows. If you don't know this in advance, then you can derive it readily as follows: $$\begin{align*} m_Y(u) &= \sum_{y=0}^\infty e^{uy} p (1-p)^y \\ &= p \sum_{y=0}^\infty ((1-p)e^u)^y \\ &= p \cdot \frac{1}{1-(1-p)e^u}, \end{align*}$$ where the last step is the consequence of the fact that the sum is an infinite geometric series with common ratio $(1-p)e^u$. Of course, this imposes the condition $|(1-p)e^u| < 1$, otherwise the series fails to converge. Since $e^u > 0$ for all $u$, and by construction $0 < p < 1$, it follows that $m_Y(u)$ is defined if and only if $u < -\log(1-p)$. And this restriction carries over into the MGF of the negative binomial distribution.