I considered $\gamma$ the following curve:
From there, I know that
\begin{equation}
\int_{\gamma}\frac{\sqrt z\log(z)}{z^2+16}\;dz = 2\pi i\left(\operatorname{Res}\left(f(z),4i\right)+\operatorname{Res}\left(f(z),-4i\right)\right) = \frac{2\pi i}8e^{\frac{\pi i}4}(1-i)(\log(16)+\pi) = \frac{\sqrt2}4\pi i(\log(16)+\pi)
\end{equation}
\begin{equation}
\operatorname{Res}(f,4i) = \lim_{z\to 4i}(z-4i)f(z) = \lim_{z\to 4i}\frac{\sqrt z\log(z)}{x+4i} = \frac{2e^{\frac{\pi i}{4}}(\log(16)+\frac\pi2 i)}{8i} = \frac18e^{\frac{\pi i}{4}}(-i\log(16)+\pi)
\end{equation}
\begin{equation}
\operatorname{Res}(f,-4i) = \lim_{z\to -4i}(z+4i)f(z) = \lim_{z\to -4i}\frac{\sqrt z\log(z)}{x-4i} = \frac{-2e^{\frac{3\pi i}{4}}(\log(16)-\frac\pi2 i)}{-8i} = \frac18e^{\frac{\pi i}{4}}(\log(16)-\pi i)
\end{equation}
We also have that
\begin{equation}
\int_{\gamma}f(z)\;dz = \int_{\gamma_1}f(z)\;dz + \int_{\gamma_2}f(z)\;dz – \int_{\gamma_3}f(z)\;dz – \int_{\gamma_4}f(z)\;dz
\end{equation}
It is easy to check that $\int_{\gamma_2}f(z)\;dz\to 0$ when $R\to\infty$ and $\int_{\gamma_4}f(z)\;dz\to 0$ when $\varepsilon\to0^+$. Then, the only thing we need to do is the following:
\begin{equation}
\int_{\gamma_1}f(z)\;dz = \int_0^{\sqrt{R^2-\varepsilon^2}}f(x+i\varepsilon)\;dx\to \int_0^{\infty}f(x)\;dx\text{ when $R\to\infty$ and $\varepsilon\to0^+$}
\end{equation}
Now, with $\gamma_3$ we get that
\begin{equation}
\int_{\gamma_3}f(z)\;dz = \int_0^{\sqrt{R^2-\varepsilon^2}}f(x-i\varepsilon)\;dx\to -\int_0^{\infty}\frac{\sqrt x(\log(x) + 2\pi i)}{x^2+16}\;dx\text{ when $R\to\infty$ and $\varepsilon\to0^+$}
\end{equation}
Therefore:
\begin{equation}
\frac{\sqrt2}4\pi i(\log(16)+\pi) = 2\int_0^{\infty}\frac{\sqrt x\log(x)}{x^2+16}\;dx + \int_0^{\infty}\frac{2\pi i\sqrt x}{x^2+16}\;dx\;\;\;\;(1)
\end{equation}
To not complicate this much, I computed $\int_0^{\infty}\frac{2\pi i\sqrt x}{x^2+16}\;dx$ in Mathematica and got $\frac{i\pi^2}{\sqrt2}$. Substracting it in (1) we get
\begin{equation}
\int_0^{\infty}\frac{\sqrt x\log(x)}{x^2+16}\;dx = i\pi\frac{\log(16)-\pi}{4\sqrt2}\ne \pi\frac{\log(16)+\pi}{4\sqrt2}
\end{equation}
which is the result that Mathematica gives me.
Could anyone please check where my calculations are wrong?
Best Answer
Care must be taken to chose the correct branch of square roots and logarithms. For example, with the branch cut at the positive real axis we have $$ \log(-4i) = \log 4 + \frac{3\pi}{2}i \, . $$
I get the following residues: $$ 2\pi i \operatorname{Res}(f(z), 4i) = 2 \pi i \frac{\sqrt{4i}\log(4i)}{8i} = \frac{\pi}{2\sqrt 2}(1+i)(\log 4 + \frac{\pi}{2}i) \\ = \frac{\pi}{2\sqrt 2}(\log 4 - \frac{\pi}{2} + I_1) $$ and $$ 2\pi i \operatorname{Res}(f(z), -4i) = 2 \pi i \frac{\sqrt{-4i}\log(-4i)}{-8i} = \frac{\pi}{-2\sqrt 2}(-1+i)(\log 4 + \frac{3\pi}{2}i) = \frac{\pi}{2\sqrt 2}(\log 4 + \frac{3\pi}{2} + I_2) $$ where $I_1$ and $I_2$ are purely imaginary numbers. It follows that $$ \tag{*} \frac{\pi}{2\sqrt 2}(2 \log 4 + \pi + I_1 + I_2) = 2\int_0^{\infty}\frac{\sqrt x\log(x)}{x^2+16}\;dx + \int_0^{\infty}\frac{2\pi i\sqrt x}{x^2+16}\;dx $$ and taking real part gives the expected result $$ \int_0^{\infty}\frac{\sqrt x\log(x)}{x^2+16}\;dx = \frac{\pi}{4\sqrt 2}( \log (16) + \pi) \, . $$
Note also that the explicit value of the second integral in $(*)$ is not needed since it is purely imaginary.