J. Borwein's review on experimental mathematics gives the following
$$\int_0^{\infty } \frac{\tan ^{-1}\left(\sqrt{a^2+x^2}\right)}{\left(x^2+1\right)\sqrt{a^2+x^2}} \, dx=\frac{\pi \left(2 \tan ^{-1}\left(\sqrt{a^2-1}\right)-\tan^{-1}\left(\sqrt{a^4-1}\right)\right)}{2 \sqrt{a^2-1}}, \ a>1$$
How can we establish it? Any help will be appreciated.
Update: The original problem has a typo and the current one is easy.
Best Answer
As suggested by Ali Shather in the comments, we can write the inverse tangent as an integral to obtain \begin{align} \int \limits_0^\infty \frac{\arctan\left(\sqrt{a^2+x^2}\right)}{(1+x^2)\sqrt{a^2+x^2}} \, \mathrm{d} x &= \int \limits_0^\infty \int \limits_0^1 \frac{\mathrm{d} y}{1+y^2 (a^2+x^2)} \,\frac{\mathrm{d} x}{1+x^2} \\ &= \int \limits_0^1 \int \limits_0^\infty \frac{\mathrm{d} x}{(1+x^2)(1+y^2(a^2+x^2))} \, \mathrm{d} y \\ &= \int \limits_0^1 \frac{1}{1 + (a^2-1)y^2} \int \limits_0^\infty \left(\frac{1}{1+x^2} - \frac{1}{a^2+\frac{1}{y^2} + x^2}\right) \, \mathrm{d} x \, \mathrm{d} y \\ &= \frac{\pi}{2} \int \limits_0^1 \frac{\mathrm{d} y}{\sqrt{1+a^2 y^2}(y + \sqrt{1+a^2 y^2})} \\&\!\!\!\!\!\!\stackrel{y = \frac{2t}{a(1-t^2)}}{=} \pi \int \limits_0^{\frac{a}{1+\sqrt{1+a^2}}}\frac{\mathrm{d} t}{a(1+t^2)+2t} \\ &= \frac{\pi a}{a^2 - 1} \int \limits_0^{\frac{a}{1+\sqrt{1+a^2}}} \frac{\mathrm{d}t}{1 + \left(\frac{1+at}{\sqrt{a^2 - 1}}\right)^2} \\ &\!\!\!\!\!\!\!\stackrel{\frac{1+at}{\sqrt{a^2 - 1}} = \frac{1}{u}}{=} \frac{\pi}{\sqrt{a^2-1}} \int \limits_{\sqrt{\frac{a^2-1}{a^2+1}}}^{\sqrt{a^2-1}} \frac{\mathrm{d} u}{1+u^2} \\ &= \frac{\pi}{\sqrt{a^2-1}}\left[\arctan\left(\sqrt{a^2-1}\right) - \arctan\left(\sqrt{\frac{a^2-1}{a^2+1}}\right)\right] \\ &= \frac{\pi}{\sqrt{a^2-1}}\left[\arctan\left(\sqrt{a^2-1}\right) - \frac{1}{2}\arctan\left(\sqrt{a^4-1}\right)\right] \end{align} for $a > 1$. The last step follows from $\arctan(x) = \frac{1}{2} \arctan\left(\frac{2 x}{1-x^2}\right)$ for $x^2 < 1$. Using the more general argtangent addition formula this result can also be written as $$ \int \limits_0^\infty \frac{\arctan\left(\sqrt{a^2+x^2}\right)}{(1+x^2)\sqrt{a^2+x^2}} \, \mathrm{d} x = \frac{\pi}{\sqrt{a^2 - 1}} \arctan \left(\frac{\sqrt{a^2-1}}{2 + \sqrt{a^2+1}}\right) \, , \, a > 1 \, .$$