You use an almost-keyhole contour, except that you indent both paths above and below the real axis with a small semicircle to avoid the pole at $z=1$:
In doing this, you end up with not $4$, but $8$ contour segments. I will avoid writing them all out by noting that the integrals over the outer circular arc and inner circular arc at the origin vanish in the limits of their radii going to $\infty$ and $0$, respectively. We are left with
$$\oint_C dz \frac{z^{-a}}{1-z} = \int_{\epsilon}^{1-\epsilon} dx \frac{x^{-a}}{1-x} + i \epsilon \int_{\pi}^0 d\phi\, e^{i \phi} \frac{(1+\epsilon e^{i \phi})^{-a}}{-\epsilon e^{i \phi}} + \int_{1+\epsilon}^{\infty} dx \frac{x^{-a}}{1-x} \\+e^{-i 2 \pi a} \int_{\infty}^{1+\epsilon} dx \frac{x^{-a}}{1-x} +e^{-i 2 \pi a} i \epsilon \int_{2 \pi}^{\pi} d\phi\, e^{i \phi} \frac{(1+\epsilon e^{i \phi})^{-a}}{-\epsilon e^{i \phi}} +e^{-i 2 \pi a} \int_{1-\epsilon}^{\epsilon} dx \frac{x^{-a}}{1-x} $$
Combining like terms, we get
$$\oint_C dz \frac{z^{-a}}{1-z} = \left ( 1-e^{-i 2 \pi a}\right ) PV\int_{0}^{\infty} dx \frac{x^{-a}}{1-x} + \left ( 1+e^{-i 2 \pi a}\right ) i \pi = 0$$
because of Cauchy's Theorem. $PV$ denotes the Cauchy principal value. After a little algebra, the result is
$$PV\int_{0}^{\infty} dx \frac{x^{-a}}{1-x} = -i \pi \frac{1+e^{-i 2 \pi a}}{1-e^{-i 2 \pi a}}=-\pi \cot{\pi a}$$
EXAMPLE
Let's check the result for $a=1/2$. This would imply that
$$PV \int _{0}^{\infty} dx \frac{1}{\sqrt{x} (1-x)} = 0$$
Consider
$$\begin{align}\underbrace{\int_0^{1-\epsilon} dx \frac{1}{\sqrt{x} (1-x)}}_{x=1/u} &= \int_{1/(1-\epsilon)}^{\infty} \frac{du}{u^2} \frac{\sqrt{u}}{1-(1/u)} \\ &= -\int_{1+\epsilon}^{\infty} du \frac{1}{\sqrt{u} (1-u)}\end{align}$$
Thus
$$\int_0^{1-\epsilon} dx \frac{1}{\sqrt{x} (1-x)} + \int_{1+\epsilon}^{\infty} du \frac{1}{\sqrt{u} (1-u)} = 0$$
or
$$PV \int _{0}^{\infty} dx \frac{1}{\sqrt{x} (1-x)} = 0$$
as was to be demonstrated.
As @Adrian suggested, define $\log z =\log |z|+i\arg(z)$ where $\arg(z)\in (0,2\pi)$ and let the contour be a keyhole contour.
Then
$$
\left|\int_{\gamma_R}\frac{e^{(1/2+i)\log z}}{z^2+1}\,dz\right|\le \int_{\gamma_R}\frac{e^{1/2 \log|z|-\arg(z)}}{R^2-1}\,|dz|\le C\frac{R^{3/2}}{R^2-1}\stackrel{R\to\infty}\longrightarrow 0,
$$
$$
\left|\int_{\gamma_r}\frac{e^{(1/2+i)\log z}}{z^2+1}\,dz\right|\le \int_{\gamma_r}\frac{e^{1/2 \log|z|-\arg(z)}}{1-r^2}\,|dz|\le Cr^{3/2}\stackrel{r\to 0}\longrightarrow 0.
$$ Thus it follows by residue theorem
$$
\lim_{\epsilon\to 0}\left(\int_{\gamma_\epsilon} f(z)dz +\int_{\gamma_{-\epsilon}} f(z)dz\right) =2\pi i\left(\text{res}_{z=i}f(z)+\text{res}_{z=-i}f(z)\right).
$$
We find$$
\lim_{\epsilon\to 0}\int_{\gamma_\epsilon} f(z)dz=\int_0^\infty \frac{\sqrt{x}e^{i\ln x}}{x^2+1}\,dx,
$$
$$
\lim_{\epsilon\to 0}\int_{\gamma_{-\epsilon}} f(z)dz=-\int_0^\infty \frac{e^{(1/2+i)(\ln x+2\pi i)}}{x^2+1}\,dx=+e^{-2\pi}\int_0^\infty \frac{\sqrt{x}e^{i\ln x}}{x^2+1}\,dx.
$$ And also
$$
\text{res}_{z=i}f(z)=\frac{e^{(1/2+i)\frac{\pi i}{2}}}{2i}=\frac{e^{-\pi/2+\pi i/4}}{2i},
$$
$$
\text{res}_{z=-i}f(z)=-\frac{e^{(1/2+i)\frac{3\pi i}{2}}}{2i}=-\frac{e^{-3\pi/2+3\pi i/4}}{2i}.
$$
Thus the given integral is
$$
\frac{\pi}{1+e^{-2\pi}}\Re\left(e^{-\pi/2+\pi i/4}-e^{-3\pi/2+3\pi i/4}\right)=\frac{\pi\cosh(\frac{\pi}{2})}{\sqrt{2}\cosh(\pi)}\sim 0.4805.
$$
(I found that this value coincides with the integral numerically by wolframalpha.)
Best Answer
Consider the integral
$$\oint_C dz \frac{\log^2{z}}{(1+z)^3}$$
where $C$ is a keyhole contour in the complex plane, about the positive real axis. This contour integral may be seen to vanish along the outer and inner circular contours about the origin, so the contour integral is simply equal to
$$\int_0^{\infty} dx \frac{\log^2{x}-(\log{x}+i 2 \pi)^2}{(1+x)^3} = -i 4 \pi \int_0^{\infty} dx \frac{\log{x}}{(1+x)^3}+4 \pi^2 \int_0^{\infty} dx \frac{1}{(1+x)^3}$$
By the residue theorem, the contour integral is also equal to $i 2 \pi$ times the residue at the pole $z=-1=e^{i \pi}$. In this case, with the triple pole, we have the residue being equal to
$$\frac12 \left [ \frac{d^2}{dz^2} \log^2{z}\right]_{z=e^{i \pi}} = 1-i \pi$$
Thus we have that
$$-i 4 \pi \int_0^{\infty} dx \frac{\log{x}}{(1+x)^3}+4 \pi^2 \frac12 = i 2 \pi + 2 \pi^2$$
which implies that
$$\int_0^{\infty} dx \frac{\log{x}}{(1+x)^3} = -\frac12$$