Let $\gamma$ be the path along the real axis then circling back counter-clockwise through the upper half-plane, letting the circle get infinitely big.
$$
\begin{align}
\int_{-\infty}^\infty\frac{\cos(x)}{x^2+a^2}\mathrm{d}x\tag{1}
&=\Re\left(\int_{-\infty}^\infty\frac{\exp(ix)}{x^2+a^2}\mathrm{d}x\right)\\\tag{2}
&=\Re\left(\int_{\gamma}\frac{\exp(ix)}{x^2+a^2}\mathrm{d}x\right)\\\tag{3}
&=\Re\left(2\pi i\,\mathrm{Res}\left(\frac{\exp(ix)}{x^2+a^2},ia\right)\right)\\\tag{4}
&=\Re\left(2\pi i\,\lim_{z\to ia}\frac{\exp(ix)}{x+ia}\right)\\\tag{5}
&=\Re\left(2\pi i\,\frac{\exp(-a)}{2ia}\right)\\[3pt]
&=\frac{\pi \exp(-a)}{a}\tag6
\end{align}
$$
$(1)$: $\Re(\exp(ix))=\cos(x)$
$(2)$: The integral along the circle back through the upper half-plane vanishes as the circle gets bigger.
$(3)$: There is only one singularity of $\dfrac{\exp(ix)}{x^2+a^2}$ in the upper half-plane, at $x=ia$. The integral along $\gamma$ is the residue of $\dfrac{\exp(ix)}{x^2+a^2}$ at $x=ia$.
$(4)$: The singularity of $\dfrac{\exp(ix)}{x^2+a^2}$ at $x=ia$ is a simple pole. We can compute the residue as $\displaystyle\lim_{x\to ia}(x-ia)\frac{\exp(ix)}{x^2+a^2}=\lim_{x\to ia}\frac{\exp(ix)}{x+ia}$.
$(5)$: plug in $x=ia$.
$(6)$: evaluate
Following the same strategy,
$$
\begin{align}
\int_{-\infty}^\infty\frac{x\sin(x)}{x^2+a^2}\mathrm{d}x
&=\Im\left(\int_{-\infty}^\infty\frac{x\exp(ix)}{x^2+a^2}\mathrm{d}x\right)\\
&=\Im\left(\int_{\gamma}\frac{x\exp(ix)}{x^2+a^2}\mathrm{d}x\right)\\
&=\Im\left(2\pi i\,\mathrm{Res}\left(\frac{x\exp(ix)}{x^2+a^2},ia\right)\right)\\
&=\Im\left(2\pi i\,\lim_{z\to ia}\frac{x\exp(ix)}{x+ia}\right)\\
&=\Im\left(2\pi i\,\frac{ia\exp(-a)}{2ia}\right)\\[6pt]
&=\pi \exp(-a)
\end{align}
$$
$\def\th{\theta}
\def\p{\pi}
\def\e{\varepsilon}
\def\g{\gamma}
\def\G{\Gamma}
\DeclareMathOperator{\sech}{sech}$Let
\begin{align*}
z &= r_1 e^{i\th_1} \\
&= 1+r_2 e^{i\th_2}
\end{align*}
where $r_i>0$ and $0\le\th_i<2\p$.
(Here we choose the branch cuts for the two roots and, with the restriction on $\th_i$, are examining the principal branch.)
Then
\begin{align*}
g(z) &= \frac{1}{\sqrt{z(1-z)}} \\
&= \frac{1}{\sqrt{r_1 r_2}} e^{-i(\th_1+\th_2+\p)/2}.
\end{align*}
Let $0<\e\ll 1$.
It is a straightforward exercise to show that the branch cut for $g(z)$ extends along the real line from $x=0$ to $x=1$ and that
\begin{align*}
\lim_{\e\to 0^+} g(x+i\e) &= -\lim_{\e\to 0^+} g(x-i\e)
= -\frac{1}{\sqrt{x(1-x)}}
& \textrm{for }0<x<1.
\end{align*}
Let $\g = \sum_{i=1}^4 \g_i$ be the counterclockwise contour with
\begin{align*}
\g_1 &: t-i\e,
t\in(0,1) \\
\g_2 &: 1+\e e^{i\th},
\th\in[-\p/2,\p/2] \\
\g_3 &: 1-t+i\e,
t\in(0,1) \\
\g_4 &: \e e^{i\th},
\th\in[\p/2,3\p/2].
\end{align*}
One can show that $\int_{\g_2} = \int_{\g_4} = 0$ in the limit $\e\to 0^+$.
(Note that, $\int_{\g_2}, \int_{\g_4}\sim \sqrt\e$.)
Thus, in the limit $\e\to 0^+$,
\begin{align*}
\int_\g &\to \int_{\g_1}+\int_{\g_3} \\
&= \int_0^1 g(t-i\e)dt + \int_0^1 g(1-t+i\e)dt \\
&= \int_0^1 g(t-i\e)dt - \int_0^1 g(t+i\e)dt \\
&\to 2\int_0^1 \frac{dx}{\sqrt{x(1-x)}}.
\end{align*}
Thus,
$$\int_0^1 \frac{dx}{\sqrt{x(1-x)}} = \frac 1 2 \int_\g g(z)dz.$$
Now deform the contour to $\G: R e^{i\th}, \th\in[0,2\p)$, where $R>1$, and consider the limit $R\to\infty$.
One can show that, for large $|z|$,
$g(z) = -i/z + O(1/z^2)$
and so
$\int_\G g(z) dz = 2\p i(-i) = 2\pi$
in the limit $R\to\infty$.
Thus,
$$\int_0^1 \frac{dx}{\sqrt{x(1-x)}} = \p.$$
Avoiding branch cuts
Let $x = \frac{1}{2}(1+\tanh t)$.
Then
$$\int_0^1 \frac{dx}{\sqrt{x(1-x)}} = \int_{-\infty}^\infty \sech t\,dt.$$
In the spirit of the question we evaluate this integral using residue calculus.
Note that
\begin{align*}
\int_{-\infty}^\infty \sech t\,dt
&= \lim_{\e\to 0^+}\int_{-\infty}^\infty e^{i\e t}\sech t\,dt \\
&= \lim_{\e\to 0^+}\int_\g e^{i\e z}\sech z\,dz,
\end{align*}
where $\g$ encircles the poles in the upper half plane in a counterclockwise manner.
The (simple) poles occur at $z = (2n+1)\p i/2$ for $n=0,1,\ldots$.
It is a straightforward exercise to find the residues,
$$\mathrm{Res}_{z=(2n+1)\p i/2} e^{i\e z}\sech z
= (-1)^{n+1}i e^{-(2n+1)\p \e/2}.$$
Then
\begin{align*}
\sum \mathrm{Res}\, e^{i\e z}\sech z
&= \sum_{n=0}^\infty (-1)^{n+1}i e^{-(2n+1)\p \e/2} \\
&= -i e^{-\p\e/2} \sum_{n=0}^\infty (-e^{-\p\e})^n \\
&= -\frac{i e^{\p\e/2}}{1+e^{\p\e}} \\
&\to -i/2.
\end{align*}
Thus,
$$\int_0^1 \frac{dx}{\sqrt{x(1-x)}} = 2\pi i(-i/2) = \p.$$
Best Answer
There are a number of ways to attack this using the residue theorem. One way is to consider the following contour integral:
$$\oint_C dz \frac{\log{z}}{1+z^4} $$
where $C$ is a quarter circle of radius $R$ in the upper-right quadrant, modified by a small quarter-circle of radius $\epsilon$ so as to avoid the branch point at $z=0$. Thus the contour integral is equal to
$$\int_{\epsilon}^R dx \frac{\log{x}}{1+x^4} + i R \int_0^{\pi/2} d\theta \, e^{i \theta} \frac{\log{(R e^{i \theta})}}{1+R^4 e^{i 4 \theta}} \\ + i \int_R^{\epsilon} dy \, \frac{\log{y} + i \pi/2}{1+y^4} + i \epsilon \int_{\pi/2}^0 d\phi \, e^{i \phi} \frac{\log{(\epsilon e^{i \phi})}}{1+\epsilon^4 e^{i 4 \phi}}$$
As $R\to\infty$, the second integral vanishes as $\log{R}/R^3$; as $\epsilon \to 0$, the fourth integra. vanishes as $\epsilon \log{\epsilon}$. In these limits, the contour integral becomes
$$(1-i) \int_0^{\infty} dx \frac{\log{x}}{1+x^4} + \frac{\pi}{2} \int_0^{\infty} \frac{dx}{1+x^4}$$
By the residue theorem, the contour integral is $i 2 \pi$ times the residue of the pole inside $C$, namely $e^{i \pi/4}$, so that we have
$$(1-i) \int_0^{\infty} dx \frac{\log{x}}{1+x^4} + \frac{\pi}{2} \int_0^{\infty} \frac{dx}{1+x^4} = i 2 \pi \frac{i \pi/4}{4 e^{i 3 \pi/4}} = \frac{\pi^2}{8} e^{i \pi/4} $$
By equating imaginary parts, we find that
$$\int_0^{\infty} dx \frac{\log{x}}{1+x^4} = -\frac{\sqrt{2}}{16} \pi^2 $$
We can also find the other integral as well from the real part:
$$\int_0^{\infty} \frac{dx}{1+x^4} = \frac{\sqrt{2}}{4} \pi $$