Define the function $I(s)$ for $s > 0$ by
$$ I(s) = \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - e^{s}\sqrt{x+1} )}{x+1} \, dx. $$
By observing that $I(\infty) = 0$, we have
$$ I(s) = -\int_{s}^{\infty} I'(t) \, dt. $$
Applying Leibniz's integral rule,
$$ I'(s) = \int_{0}^{\infty} \frac{e^{s}\sqrt{x+1}}{1 + (\sqrt{x} - e^{s}\sqrt{x+1} )^{2}} \, \frac{dx}{x+1}. $$
Now with the substitution $x = \tan^{2}\theta$,
\begin{align*}
I'(s)
&= \int_{0}^{\frac{\pi}{2}} \frac{\sin\theta}{\cosh s - \sin\theta}
= \frac{1}{2} \int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \frac{\cos\theta}{\cosh s - \cos\theta}.
\end{align*}
Then with the substitution $z = e^{i\theta}$, it follows that
\begin{align*}
I'(s)
&= \frac{i}{2} \int_{\Gamma} \frac{z^{2} + 1}{z^{2} - 2z \cosh s + 1} \frac{dz}{z},
\end{align*}
where the contour $\Gamma$ denotes the counter-clockwise semicircular arc joining from $-i$ to $i$. Note that we have $z^{2} - 2z \cosh s + 1 = 0$ if and only if $\cosh s = \cos \theta$ if and only if $z = e^{i\theta} = e^{\pm s}$. Thus deforming the contour to the straight line joining from $-i$ to $i$ with infinitesimal indent at the origin, we obtain
\begin{align*}
I'(s)
&= \frac{i}{2} \left\{ \operatorname{PV}\! \int_{-i}^{i} f(z) \, dz + 2\pi i \operatorname{Res} (f, e^{-s}) + \pi i \operatorname{Res} (f, 0) \right\},
\end{align*}
where $f$ denotes the integrand
$$ f(z) = \frac{z^{2} + 1}{z (z^{2} - 2z \cosh s + 1)}. $$
Proceeding the calculation,
\begin{align*}
I'(s)
&= \frac{i}{2} \left\{ i \operatorname{PV} \! \int_{-1}^{1} f(ix) \, dx + \pi i - 2 \pi i \coth s \right\} \\
&= - \frac{1}{2} \int_{0}^{1} \{ f(ix) + f(-ix) \} \, dx - \frac{\pi}{2} + \pi \coth s \\
&= \cosh s \int_{0}^{1} \frac{\frac{1}{2} (1 - x^{-2}) }{\{ \frac{1}{2}(x + x^{-1}) \}^{2} + \sinh^{2} s} \, dx - \frac{\pi}{2} + \pi \coth s.
\end{align*}
Now with the substitution $u = \frac{1}{2} \{ x + x^{-1} \}$, it follows that
\begin{align*}
I'(s)
&= - \cosh s \int_{1}^{\infty} \frac{du}{u^{2} + \sinh^{2} s} - \frac{\pi}{2} + \pi \coth s \\
&= - \arctan(\sinh s) \coth s - \frac{\pi}{2} + \pi \coth s.
\end{align*}
Finally, with the substitution $x = \sinh t$,
\begin{align*}
I(s)
= - \int_{s}^{\infty} I'(t) \, dt
&= \int_{s}^{\infty} \left\{ \arctan(\sinh t) \coth t + \frac{\pi}{2} - \pi \coth t \right\} \, dt \\
&= \int_{\sinh s}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx.
\end{align*}
Our problem corresponds to $s = \log 2$, or equivalently, $a := \sinh s = \frac{3}{4}$.
\begin{align*}
&\int_{a}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx \\
&= - \int_{a}^{\infty} \frac{\arctan (1/x)}{x} \, dx + \frac{\pi}{2} \int_{a}^{\infty} \left( \frac{1}{\sqrt{x^{2} + 1}} - \frac{1}{x} \right) \, dx \\
&= - \int_{0}^{1/a} \frac{\arctan x}{x} \, dx + \frac{\pi}{2} \lim_{R\to\infty} \left[ \log\left( \frac{x + \sqrt{x^{2} + 1}}{x} \right) \right]_{a}^{R} \\
&= - \operatorname{Ti}\left(\frac{1}{a}\right) - \frac{\pi}{2} \log\left( \frac{a + \sqrt{a^{2} + 1}}{2a} \right),
\end{align*}
where the function
$$ \operatorname{Ti}(x) = \frac{\operatorname{Li}_{2}(ix) - \operatorname{Li}_{2}(-ix)}{2i} $$
is the inverse tangent integral. Using the following simple identity
$$ \operatorname{Ti}(x) = \operatorname{Ti}\left(\frac{1}{x}\right) + \frac{\pi}{2}\log x, $$
it follows that
\begin{align*}
\int_{a}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx
= - \operatorname{Ti}(a) - \frac{\pi}{2} \log\left( \frac{a + \sqrt{a^{2} + 1}}{2a^{2}} \right).
\end{align*}
Therefore, plugging $a = \frac{3}{4}$ gives
$$ \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - 2\sqrt{x+1} )}{x+1} \, dx
= \pi \log(3/4) - \operatorname{Ti}(3/4) $$
as Cleo pointed out without explanation. More generally, for $k > 1$
$$ \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - k \sqrt{x+1} )}{x+1} \, dx
= \frac{\pi}{2} \log \left( \frac{(k^{2} - 1)^{2}}{2k^{3}} \right) - \operatorname{Ti}\left( \frac{k^{2} - 1}{2k} \right). $$
Best Answer
Expand $\tan^{-1}(y)$ via its logarithm definition:
$$\int_0^\pi\arctan(1+\cos x)dx=\frac i2\int_0^\pi\ln\left(1+\frac{1-i}2\cos(x)\right)-\ln\left(1+\frac{1+i}2\cos(x)\right)-\frac{\pi i}2dx$$
Now apply:
and the principal logarithm:
$$\frac i2\int_0^\pi\ln\left(1+\frac{1-i}2\cos(x)\right)-\ln\left(1+\frac{1+i}2\cos(x)\right)-\frac{\pi i }2dx=\frac{\pi i}2\ln(-i)+\frac{\pi i}2\ln\left(\frac{\sqrt{4+2i}+2}{\sqrt{4-2i}+2}\right)=\frac{\pi i} 2\ln\left(\sqrt5–2-2\sqrt{\sqrt5-2}i\right)$$
We rationalized to get the last expression. Finally use $\frac i2\ln(x)=\cot^{-1}(i\frac{x+1}{1-x})$ and rationalize again to get:
$$\int_0^\pi\arctan(1+\cos x)dx=\pi\cot^{-1}\left(\sqrt{\frac{\sqrt5+1}2}\right)=\pi\cot^{-1}(\sqrt\phi)$$