Let $k\neq 0$, $y\neq 0$ be real numbers and let $0<\alpha<1$.
Then, how can I evaluate the integral
$$\int_{-\infty}^\infty \frac{e^{ikx}}{(ix)^\alpha (i(x+y))^\alpha} dx \; ?$$
Note that the integral converges as the improper Riemann integral. We interpret $(ix)^\alpha$ and $(i(x+y))^\alpha$ to be in the principal branch, where the argument is in $(-\pi, \pi]$.
Mathematica cannot evaluate this integral.
Best Answer
Let us assume $y \in \mathbb{R}^{+}$. For $x<0$,
\begin{align*} (\mathrm{i}x)^\alpha &= (-\mathrm{i}u), \\ \left[ \mathrm{i}(x+y)\right]^\alpha &= \left[ -\mathrm{i}(u-y)\right]^\alpha \text{for } u > y, \text{where } u\equiv -x \text{ (therefore } u \in \mathbb{R}^{+}\text{)}. \end{align*}
Now let
\begin{align*} J &= \int_{-\infty}^\infty \frac{e^{ikx}}{(ix)^\alpha (i(x+y))^\alpha} dx \\ &= \int_{y}^{\infty}\frac{e^{-iku}}{(-iu)^\alpha \left[-i(u-y)\right]^\alpha} du + \int_{0}^{y}\frac{e^{-iku}}{(-iu)^\alpha \left[ i(y-u)\right]^\alpha} du + \int_{0}^\infty \frac{e^{ikx}}{(ix)^\alpha (i(x+y))^\alpha} dx\\ &= e^{i\pi\alpha}\int_{y}^{\infty}\frac{e^{-iku}}{u^\alpha \left[(u-y)\right]^\alpha} du + \int_{0}^{y}\frac{e^{-iku}}{u^\alpha \left[ (y-u)\right]^\alpha} du + \int_{0}^\infty \frac{e^{ikx}}{(ix)^\alpha (i(x+y))^\alpha} dx\\ &= J_{1} + J_{2} + J_{3} \end{align*}
For the evaluation of $J_{1}$, let $t\equiv (u-y)^{1-\alpha}$. Then we have
\begin{align*} J_{1} &= e^{i\pi\alpha}\int_{y}^{\infty}\frac{e^{-iku}}{u^\alpha \left[(u-y)\right]^\alpha} du \\ &= \gamma e^{i\pi\alpha}\int_{0}^{\infty}\frac{e^{-ik(t^\gamma + y)}}{(t^\gamma + y)^\alpha} dt \end{align*}
where $\gamma \equiv 1/(1-\alpha)$.
Mathematica gives $J_{1}$ as
\begin{align*} J_{1} =\frac{y^{-2 \alpha } e^{i (\pi \alpha -k y)}}{8 \sqrt{\pi } k} \left(i 4^{\alpha } k^2 y^2 \Gamma (1-\alpha ) \Gamma \left(\alpha -\frac{1}{2}\right) \\ \, _2F_3\left(1-\frac{\alpha }{2},\frac{3}{2}-\frac{\alpha }{2}; \frac{3}{2},\frac{3}{2}-\alpha ,2-\alpha ;-\frac{1}{4} k^2 y^2\right)+\\4 \sqrt{\pi } \Gamma (1-2 \alpha ) (k y)^{\alpha } \\ \left(e^{i \pi \alpha } (k y)^{\alpha +1} \, _2F_3\left(\frac{\alpha }{2}+\frac{1}{2},\frac{\alpha }{2}+1;\frac{3}{2},\alpha +\frac{1}{2},\alpha +1;-\frac{1}{4} k^2 y^2\right)-\\4^{\alpha } \Gamma \left(\alpha +\frac{1}{2}\right) \sqrt{k y} \cos \left(\frac{k y}{2}\right) \left(J_{\frac{1}{2}-\alpha }\left(\frac{k y}{2}\right)+i e^{i \pi \alpha } J_{\alpha -\frac{1}{2}}\left(\frac{k y}{2}\right)\right)\right)\right) \end{align*}
where $\, _pF_q(a;b;z)$ is the generalized hypergeometric function.
For $k=y=1$, $\lim_{\alpha \to 1/2} J_{1} \approx 1.62844 - 0.0939721i$ (same as GEdgar's).
Also, for $k=y=1,\alpha=1/2$, Mathematica gives
\begin{align*} \int_{-\infty}^{-1} \frac{e^{ix}}{(ix)^{1/2} (i(x+1))^{1/2}} dx= \frac{\pi}{\sqrt{2}} \left(G_{2,4}^{3,0}\left(\frac{1}{4}| \begin{array}{c} \frac{1}{4},\frac{3}{4} \\ 0,\frac{1}{2},\frac{1}{2},0 \\ \end{array} \right)+i G_{2,4}^{3,0}\left(\frac{1}{4}| \begin{array}{c} \frac{1}{4},\frac{3}{4} \\ 0,0,\frac{1}{2},\frac{1}{2} \\ \end{array} \right)\right) \end{align*}
where $G$ is the MeijerG-function. This last function numerically evaluates as $J_1$ for for $k=y=1, \alpha=1/2$.
Gradshteyn & Ryshik give the real and imaginary parts of $J_{2}$ as
\begin{align*} Re[J_2] = \frac{1}{2} B(1-\alpha ,1-\alpha ) y^{1-2 \alpha } (\, _1F_1(1-\alpha ;2-2 \alpha ;-i k y)+\, _1F_1(1-\alpha ;2-2 \alpha ;i k y)) \end{align*}
and
\begin{align*} Im[J_2] = -\frac{1}{2} i B(1-\alpha ,1-\alpha ) y^{1-2 \alpha } (\, _1F_1(1-\alpha ;2-2 \alpha ;i k y)-\, _1F_1(1-\alpha ;2-2 \alpha ;-i k y)) \end{align*}
where $B(a,b)$ is the Euler beta function and $\, _1F_1(a;b;z)$ is the Kummer confluent hypergeometric function. Therefore,
\begin{align*} J_{2} = \frac{1}{2} B(1-\alpha ,1-\alpha ) y^{1-2 \alpha } (\, _1F_1(1-\alpha ;2-2 \alpha ;i k y)-\, _1F_1(1-\alpha ;2-2 \alpha ;-i k y))+\frac{1}{2} B(1-\alpha ,1-\alpha ) y^{1-2 \alpha } (\, _1F_1(1-\alpha ;2-2 \alpha ;-i k y)+\, _1F_1(1-\alpha ;2-2 \alpha ;i k y)) \end{align*}
For $k=y=1, \alpha=1/2, J_{2} \approx 2.58737 + 1.41349i$ (the signal of the real part differs from GEdgar's $J_{2}$). In fact,
\begin{align*} \int_{-1}^{0} \frac{e^{ix}}{(ix)^{1/2} (i(x+1))^{1/2}} dx=e^{-\frac{i}{2}} \pi J_0\left(\frac{1}{2}\right) \end{align*}
Finally, Mathematica gives $J_{3}$ as
\begin{align*} J_{3} = \frac{\pi e^{-i \pi \alpha } y^{-2 \alpha }}{8 k \Gamma (\alpha ) \Gamma (2 \alpha )} \left(i \pi 4^{\alpha } k^2 y^2 \csc (2 \pi \alpha ) \Gamma (2-\alpha ) \Gamma (2 \alpha ) \\ \, _2\tilde{F}_3\left(1-\frac{\alpha }{2},\frac{3}{2}-\frac{\alpha }{2};\frac{3}{2},\frac{3}{2}-\alpha ,2-\alpha ;-\frac{1}{4} k^2 y^2\right)+2 k y \Gamma (\alpha ) (\csc (\pi \alpha )-i \sec (\pi \alpha )) (k y)^{2 \alpha } \, _2F_3\left(\frac{\alpha }{2}+\frac{1}{2},\frac{\alpha }{2}+1;\frac{3}{2},\alpha +\frac{1}{2},\alpha +1;-\frac{1}{4} k^2 y^2\right)+\\4 \sqrt{\pi } \Gamma (2 \alpha ) (k y)^{\alpha +\frac{1}{2}} \cos \left(\frac{k y}{2}\right) \left(-2 \csc (2 \pi \alpha ) J_{\frac{1}{2}-\alpha }\left(\frac{k y}{2}\right)+(\sec (\pi \alpha )+i \csc (\pi \alpha )) J_{\alpha -\frac{1}{2}}\left(\frac{k y}{2}\right)\right)\right) \end{align*}
For $k=y=1$, $\lim_{\alpha \to 1/2} J_{3} \approx 0.958926 - 1.31951i$ (same as GEdgar's $J_{3}$). For this particular choice of $k, y, \text{and } \alpha$,
\begin{align*} \int_{0}^{\infty} \frac{e^{ix}}{(ix)^{1/2} (i(x+1))^{1/2}} dx = \frac{1}{2} e^{-\frac{i}{2}} \pi H_0^{(1)}\left(\frac{1}{2}\right) \end{align*} where $H_0^{(1)}$ is the Hankel function of the first kind. This last function numerically evaluates as $J_3$ for $k=y=1, \alpha=1/2$.
Summing up,
\begin{align*} J = \frac{\pi ^{3/2} 2^{-2 \alpha -3} y^{-2 \alpha } e^{-i (\pi \alpha +k y)}}{k \Gamma (\alpha )} \left(2 e^{-\frac{1}{2} i k y} \left(2 \csc (\pi \alpha ) \sec (\pi \alpha ) e^{i (\pi \alpha +k y)} (k y)^{2 \alpha } \\ \left(\sqrt{\pi } k y \Gamma (\alpha +1) \cos \left(\frac{k y}{2}-2 \pi \alpha \right) \, _2\tilde{F}_3\left(\frac{\alpha +1}{2},\frac{\alpha +2}{2};\frac{3}{2},\alpha +\frac{1}{2},\alpha +1;-\frac{1}{4} k^2 y^2\right) \\ + 2 \, _0\tilde{F}_1\left(;\alpha +\frac{1}{2};-\frac{1}{16} k^2 y^2\right) (\sin (2 \pi \alpha )-\sin (k y-2 \pi \alpha ))\right) \\+ 16^{\alpha } k y \csc (2 \pi \alpha ) \left(-1+e^{i k y}\right) \left(e^{2 i \pi \alpha } \left(1+2 e^{i k y}\right)+e^{i k y}\right) \, _0\tilde{F}_1\left(;\frac{3}{2}-\alpha ;-\frac{1}{16} k^2 y^2\right)\right)+i \sqrt{\pi } 16^{\alpha } k^2 y^2 \csc (2 \pi \alpha ) \Gamma (2-\alpha ) \left(e^{i k y}-e^{2 i \pi \alpha }\right) \, _2\tilde{F}_3\left(1-\frac{\alpha }{2},\frac{3}{2}-\frac{\alpha }{2};\frac{3}{2},\frac{3}{2}-\alpha ,2-\alpha ;-\frac{1}{4} k^2 y^2\right)\right) \end{align*}
Plotting $J$ as a function of $\alpha$, with $k = y = 1$, yields,