Define the function $I(s)$ for $s > 0$ by
$$ I(s) = \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - e^{s}\sqrt{x+1} )}{x+1} \, dx. $$
By observing that $I(\infty) = 0$, we have
$$ I(s) = -\int_{s}^{\infty} I'(t) \, dt. $$
Applying Leibniz's integral rule,
$$ I'(s) = \int_{0}^{\infty} \frac{e^{s}\sqrt{x+1}}{1 + (\sqrt{x} - e^{s}\sqrt{x+1} )^{2}} \, \frac{dx}{x+1}. $$
Now with the substitution $x = \tan^{2}\theta$,
\begin{align*}
I'(s)
&= \int_{0}^{\frac{\pi}{2}} \frac{\sin\theta}{\cosh s - \sin\theta}
= \frac{1}{2} \int_{-\frac{\pi}{2}}^{\frac{\pi}{2}} \frac{\cos\theta}{\cosh s - \cos\theta}.
\end{align*}
Then with the substitution $z = e^{i\theta}$, it follows that
\begin{align*}
I'(s)
&= \frac{i}{2} \int_{\Gamma} \frac{z^{2} + 1}{z^{2} - 2z \cosh s + 1} \frac{dz}{z},
\end{align*}
where the contour $\Gamma$ denotes the counter-clockwise semicircular arc joining from $-i$ to $i$. Note that we have $z^{2} - 2z \cosh s + 1 = 0$ if and only if $\cosh s = \cos \theta$ if and only if $z = e^{i\theta} = e^{\pm s}$. Thus deforming the contour to the straight line joining from $-i$ to $i$ with infinitesimal indent at the origin, we obtain
\begin{align*}
I'(s)
&= \frac{i}{2} \left\{ \operatorname{PV}\! \int_{-i}^{i} f(z) \, dz + 2\pi i \operatorname{Res} (f, e^{-s}) + \pi i \operatorname{Res} (f, 0) \right\},
\end{align*}
where $f$ denotes the integrand
$$ f(z) = \frac{z^{2} + 1}{z (z^{2} - 2z \cosh s + 1)}. $$
Proceeding the calculation,
\begin{align*}
I'(s)
&= \frac{i}{2} \left\{ i \operatorname{PV} \! \int_{-1}^{1} f(ix) \, dx + \pi i - 2 \pi i \coth s \right\} \\
&= - \frac{1}{2} \int_{0}^{1} \{ f(ix) + f(-ix) \} \, dx - \frac{\pi}{2} + \pi \coth s \\
&= \cosh s \int_{0}^{1} \frac{\frac{1}{2} (1 - x^{-2}) }{\{ \frac{1}{2}(x + x^{-1}) \}^{2} + \sinh^{2} s} \, dx - \frac{\pi}{2} + \pi \coth s.
\end{align*}
Now with the substitution $u = \frac{1}{2} \{ x + x^{-1} \}$, it follows that
\begin{align*}
I'(s)
&= - \cosh s \int_{1}^{\infty} \frac{du}{u^{2} + \sinh^{2} s} - \frac{\pi}{2} + \pi \coth s \\
&= - \arctan(\sinh s) \coth s - \frac{\pi}{2} + \pi \coth s.
\end{align*}
Finally, with the substitution $x = \sinh t$,
\begin{align*}
I(s)
= - \int_{s}^{\infty} I'(t) \, dt
&= \int_{s}^{\infty} \left\{ \arctan(\sinh t) \coth t + \frac{\pi}{2} - \pi \coth t \right\} \, dt \\
&= \int_{\sinh s}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx.
\end{align*}
Our problem corresponds to $s = \log 2$, or equivalently, $a := \sinh s = \frac{3}{4}$.
\begin{align*}
&\int_{a}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx \\
&= - \int_{a}^{\infty} \frac{\arctan (1/x)}{x} \, dx + \frac{\pi}{2} \int_{a}^{\infty} \left( \frac{1}{\sqrt{x^{2} + 1}} - \frac{1}{x} \right) \, dx \\
&= - \int_{0}^{1/a} \frac{\arctan x}{x} \, dx + \frac{\pi}{2} \lim_{R\to\infty} \left[ \log\left( \frac{x + \sqrt{x^{2} + 1}}{x} \right) \right]_{a}^{R} \\
&= - \operatorname{Ti}\left(\frac{1}{a}\right) - \frac{\pi}{2} \log\left( \frac{a + \sqrt{a^{2} + 1}}{2a} \right),
\end{align*}
where the function
$$ \operatorname{Ti}(x) = \frac{\operatorname{Li}_{2}(ix) - \operatorname{Li}_{2}(-ix)}{2i} $$
is the inverse tangent integral. Using the following simple identity
$$ \operatorname{Ti}(x) = \operatorname{Ti}\left(\frac{1}{x}\right) + \frac{\pi}{2}\log x, $$
it follows that
\begin{align*}
\int_{a}^{\infty} \left( \frac{\arctan x}{x} + \frac{\pi}{2\sqrt{x^{2} + 1}} - \frac{\pi}{x} \right) \, dx
= - \operatorname{Ti}(a) - \frac{\pi}{2} \log\left( \frac{a + \sqrt{a^{2} + 1}}{2a^{2}} \right).
\end{align*}
Therefore, plugging $a = \frac{3}{4}$ gives
$$ \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - 2\sqrt{x+1} )}{x+1} \, dx
= \pi \log(3/4) - \operatorname{Ti}(3/4) $$
as Cleo pointed out without explanation. More generally, for $k > 1$
$$ \int_{0}^{\infty} \frac{\operatorname{arccot}( \sqrt{x} - k \sqrt{x+1} )}{x+1} \, dx
= \frac{\pi}{2} \log \left( \frac{(k^{2} - 1)^{2}}{2k^{3}} \right) - \operatorname{Ti}\left( \frac{k^{2} - 1}{2k} \right). $$
First I want to define with the Stirling numbers of the first kind $\left[ \begin{array}{c} n \\ k \end{array} \right]$ a special generalization of the Riemann Zeta function :
$$\zeta_n(m):=\sum\limits_{k=1}^\infty \frac{1}{k^m}\left(\frac{n!}{(k-1)!}\left[\begin{array}{c} k \\ n+1 \end{array} \right]\right)$$
and
$$\eta_n(m):=\sum\limits_{k=1}^\infty \frac{(-1)^{k-1}}{k^m}\left(\frac{n!}{(k-1)!}\left[\begin{array}{c} k \\ n+1 \end{array} \right]\right)$$
which are convergent for the integer values $\enspace m\geq 2$ .
For $\enspace n=0\enspace$ we have $\enspace\zeta_0(m)=\zeta(m)\enspace$ and $\enspace\eta_0(m)=\eta(m)\enspace$ .
Note: Obviously (because of the other results) these series can be expressed by sums of the polylogarithm function and modifications of that.
Please also see here, part Expansion by harmonic numbers, with $\enspace\displaystyle w(n,m):=\frac{m!}{(n-1)!}\left[ \begin{array}{c} n \\ {m+1} \end{array} \right]\enspace$ and it's recursion formula.
Secondly, an extension of an integral as a series, $n\in\mathbb{N}_0$ and $z\in\mathbb{R}\setminus \{2\mathbb{N}\}$ and $nz>-1$:
$ \displaystyle \int\limits_0^\pi x^n \left(2\sin\frac{x}{2}\right)^z dx=i^{-z} \int\limits_0^\pi x^n e^{i\frac{xz}{2}}(1- e^{-ix})^z dx= e^{-i\frac{\pi z}{2}} \int\limits_0^\pi x^n \sum\limits_{k=0}^\infty\binom{z}{k}(-1)^k e^{-ix(\frac{z}{2}-k)} dx$
$\displaystyle =\int\limits_0^\pi x^n e^{i(x-\pi)\frac{z}{2}} dx+ \sum\limits_{v=0}^n \frac{(-1)^v\pi^{n-v} n!}{i^{v+1}(n-v)!} \sum\limits_{k=1}^\infty \binom{z}{k}\frac{1}{(\frac{z}{2}-k)^{v+1}} $
$\displaystyle \hspace{3.5cm} -i^{n-1}n!e^{-i\frac{\pi z}{2}} \sum\limits_{k=1}^\infty \binom{z}{k}\frac{ (-1)^k}{(\frac{z}{2}-k)^{n+1}}$
using the main branch of the logarithm and therefore $\displaystyle i=e^{i\frac{\pi}{2}}$ .
The Stirling numbers of the first kind are usually defined by $\enspace \displaystyle \sum\limits_{k=0}^n \left[ \begin{array}{c} n \\ k \end{array} \right] x^k := x(x+1)…(x+n-1) $ .
Because of $\enspace \displaystyle (\sum\limits_{v=0}^\infty x^v \frac{d^k}{dz^k}\binom{z}{v}) |_{z=0} =\frac{d^k}{dz^k}(1+x)^z |_{z=0} =(\ln(1+x))^k=k!\sum\limits_{v=k}^\infty (-1)^{v-k} \left[\begin{array}{c} v \\ k \end{array} \right] \frac{x^v}{v!}$
we get $\enspace \displaystyle \binom{z}{k}|_{z=0}=0^k\enspace$ , $\enspace \displaystyle \frac{d}{dz} \binom{z}{k} |_{z=0} = (-1)^{k-1} \left[\begin{array}{c} k \\ 1 \end{array} \right] \frac{1}{k!}= \frac{(-1)^{k-1}}{k} \enspace$ , $\enspace \displaystyle \frac{d^2}{dz^2} \binom{z}{k} |_{z=0} = (-1)^{k-2} \left[\begin{array}{c} k \\ 2 \end{array} \right] \frac{2!}{k!}= \frac{(-1)^k 2}{k}\sum\limits_{j=1}^{k-1}\frac{1}{j} \enspace$ and $\enspace \displaystyle \frac{d^3}{dz^3} \binom{z}{k} |_{z=0} = (-1)^{k-3} \left[\begin{array}{c} k \\ 3 \end{array} \right] \frac{3!}{k!}= \frac{(-1)^{k-1} 3}{k}( (\sum\limits_{j=1}^{k-1}\frac{1}{j})^2 - \sum\limits_{j=1}^{k-1}\frac{1}{j^2} ) $ .
For $(n;k):=(3;3)$ follows
$\displaystyle \int\limits_0^\pi x^3 \left(\ln\left(2\sin\frac{x}{2} \right)\right)^3 dx =$
$\hspace{2cm}\displaystyle =\frac{9\pi^2}{2}\left(\zeta(5)+3\eta(5)-4\eta_1(4)+2\eta_2(3)\right) $
$\hspace{2.5cm}\displaystyle - 90\left(\zeta(7)+\eta(7)\right) +72\left(\zeta_1(6)+\eta_1(6)\right) - 18\left(\zeta_2(5)+\eta_2(5)\right) $
Note:
For the calculations I have used $\enspace\displaystyle\int\limits_0^\pi x^n e^{iax}dx = \frac{(-1)^{n+1} n!}{(ia)^{n+1}}+e^{i\pi a}\sum\limits_{v=0}^n\frac{(-1)^v \pi^{n-v}n!}{(ia)^{v+1}(n-v)!}$
with $\enspace\displaystyle a=-(\frac{z}{2}-k)$ .
And it was necessary to calculate $\enspace\displaystyle\frac{d^m}{dz^m} \binom{z}{k}\frac{1}{(\frac{z}{2}-k)^{v+1}}|_{z=0}\enspace$ and $\enspace\displaystyle\frac{d^m}{dz^m} e^{-i\frac{\pi z}{2}}\binom{z}{k}\frac{1}{(\frac{z}{2}-k)^{n+1}}|_{z=0}\enspace$ for $\enspace m\in\{0,1,2,3\}$ .
Best Answer
So, following the procedure I outlined here, I get for the transformed integral:
$$I(r,s) = \int_0^{\infty} dv \frac{4 s \left(v^2-1\right) \left(v^4-(4 r-6) v^2+1\right)}{v^8+4 \left(2 r-s^2-1\right) v^6 +2 \left(8 r^2-8 r-4 s^2+3\right) v^4 +4 \left(2 r-s^2-1\right) v^2 +1} \log{v} $$
Note that this reduces to the integral in the original problem when $r=3$ and $s=2$. Then we see that the roots of the denominator satisfy the same symmetries as before, so we need only find one root of the form $\rho e^{i \theta}$ where
$$\rho = \sqrt{\frac{r+\sqrt{r^2-s^2}}{2}} + \sqrt{\frac{r+\sqrt{r^2-s^2}}{2}-1}$$
and
$$\theta = \arctan{\sqrt{\frac{2 \left (r+\sqrt{r^2-s^2}\right )}{s^2}-1}}$$
Using the same methodology I derived, I am able to confirm your conjecture.