For parameters $k\in \mathbb{N}$ and $\lambda\in\mathbb{R}$, the density function of the gamma dist is $$f_{(k,\lambda)}(x)=\frac{x^{k-1}\lambda^ke^{-\lambda x}}{(k-1)!}$$
Then the characteristic function is
\begin{align*}
\Phi_X(\omega)&=\int_0^\infty e^{i\omega x}f_{(k,\lambda)}(x)dx\\
&=\frac{\lambda^k}{(k-1)!}\int_0^\infty x^{k-1}e^{(i\omega-\lambda)x}dx
\end{align*}
Here I applied an identity for the integral above, given as a hint in the question:
$$\frac{\lambda-i\omega}{k}\int_0^\infty x^{k}e^{(i\omega-\lambda)x}dx$$
Then I made a change of variable $(i\omega-\lambda)x=-y$, which changed the integration limits from $0$ and $\infty$, to $0$ and $-\infty$ respectively. This is where I have some doubts if it's correct. So the characteristic function came to be
$$\frac{\lambda^k}{k!(\lambda-i\omega)^k}\int_0^{-\infty}y^ke^{-y}dy$$
Now if the upper limit of the integral was $+\infty$, then the integral would be $\Gamma(k+1)=k!$ and I would've gotten the desired result. Where is it going wrong?
Best Answer
First of all, $\lambda$ must be positive for everything to make sense.
Next, as you suspect, when computing $\int_0^\infty x^{k}e^{(i\omega-\lambda)x}\,dx$, after the substitution $(i\omega-\lambda)x=-y$, the "limits" are not $0$ and $-\infty$; rather, the integration path is the ray $\{re^{i\phi} : r\geqslant 0\}$, where $\phi=\arg(\lambda-i\omega)$. To see that such an integral is equal to $\int_0^\infty$ of the same integrand, apply CIT to the sector $$\{re^{i\theta} : r\in[0,R],\theta\in[0,\phi]\};$$ as $R\to+\infty$, the integral along the circular arc part vanishes.
Alternatively, one may just note that $z\mapsto\int_0^\infty x^k e^{-zx}\,dx$ is analytic in $\Re z>0$, so that, since it is equal to $k!/z^{k+1}$ when $z$ is real (positive), the same is true for any $z$ (with $\Re z>0$) by analytic continuation.