First note that by substituting $x\mapsto kx$, we get
$$
\int_0^\infty\log\left(\frac{x^2+2kx\cos(a)+k^2}{x^2+2kx\cos(b)+k^2}\right)\frac{\mathrm{d}x}{x}
=\int_0^\infty\log\left(\frac{x^2+2x\cos(a)+1}{x^2+2x\cos(b)+1}\right)\frac{\mathrm{d}x}{x}
$$
Let $u=\frac{x+\cos(a)}{\sin(a)}$. Then
$$
\begin{align}
\frac{\mathrm{d}}{\mathrm{d}a}\int_0^\infty\log\left(\frac{x^2+2x\cos(a)+1}{x^2+2x+1}\right)\frac{\mathrm{d}x}{x}
&=-2\int_0^\infty\frac{\sin(a)}{x^2+2x\cos(a)+1}\,\mathrm{d}x\\
&=-2\int_0^\infty\frac{\sin(a)}{(x+\cos(a))^2+\sin^2(a)}\,\mathrm{d}x\\
&=-\frac2{\sin(a)}\int_0^\infty\frac1{\frac{(x+\cos(a))^2}{\sin^2(a)}+1}\,\mathrm{d}x\\
&=-2\int_{\cot(a)}^\infty\frac1{u^2+1}\,\mathrm{d}u\\[9pt]
&=-2a
\end{align}
$$
Integrating in $a$ gives
$$
\int_0^\infty\log\left(\frac{x^2+2x\cos(a)+1}{x^2+2x+1}\right)\frac{\mathrm{d}x}{x}
=-a^2
$$
Therefore, by subtraction,
$$
\int_0^\infty\log\left(\frac{x^2+2kx\cos(a)+k^2}{x^2+2kx\cos(b)+k^2}\right)\frac{\mathrm{d}x}{x}
=b^2-a^2
$$
We have the following closed form.
Proposition. $$
\int_0^1\log \left(1-x\right)\log \left(-\log x\right)\:dx=\gamma-\gamma_1+\gamma_1(1,0)\tag1
$$
where $\displaystyle \gamma$ is the Euler-Mascheroni constant, where $\gamma_1$ is the Stieltjes constant,
$$\gamma_1 = \lim_{N\to+\infty}\left(\sum_{n=1}^N \frac{\log n}n-\int_1^N\frac{\log t}t\:dt\right)$$
and where $\gamma_1(a,b)$ is the poly-Stieltjes constant (see here),
$$\gamma_1(a,b) = \lim_{N\to+\infty}\left(\sum_{n=1}^N \frac{\log (n+a)}{n+b}-\int_1^N\frac{\log t}t\:dt\right)\!.$$
Proof.
One may recall the classic integral representation of the Euler gamma function
$$
\frac{\Gamma(s)}{(a+1)^s}=\int_0^\infty t^{s-1} e^{-(a+1)t}\:dt, \qquad s>0,\, a>-1. \tag2
$$ By differentiating $(2)$ with respect to $s$, putting $s=1$ and making the change of variable $x=e^{-t}$, we get
$$
\int_0^1x^a\log\left(-\log x\right)\:dx=-\frac{\gamma+\log(a+1)}{a+1},\qquad a>-1, \tag3
$$
where $\displaystyle \gamma$ is the Euler-Mascheroni constant. We are allowed to insert the standard Taylor series expansion,
$$
\log (1-x)= -\sum_{n=1}^{\infty} \frac{x^n}n, \qquad |x|<1,\tag4
$$ into the given integral, then using $(3)$ we obtain
$$
\begin{align}
\int_0^1\log \left(1-x\right)\log \left(-\log x\right)\:dx&=-\int_0^1\sum_{n=1}^{\infty}\frac{x^n}n \:\log (-\log x)\:dx\\
&=-\sum_{n=1}^{\infty} \frac1n\int_0^1 x^n\log (-\log x)\:dx\\
&=\sum_{n=1}^{\infty} \frac{\gamma+\log(n+1)}{n(n+1)}\\
&=\gamma \sum_{n=1}^{\infty}\frac1{n(n+1)}+\sum_{n=1}^{\infty} \frac{\log(n+1)}{n(n+1)}\\
&=\gamma +\sum_{n=1}^{\infty} \frac{\log(n+1)}{n(n+1)},\tag5
\end{align}
$$ and we may conclude with Theorem $2$ here to get
$$
\begin{align}
\sum_{n=1}^{\infty} \frac{\log (n+1)}{n(n+1)}=\gamma_1({1,0})-\gamma_1,\tag6
\end{align}
$$
since $\gamma_1(1,1)=\gamma_1$.
Remark. I am inclined to believe that the poly-Stieltjes constants will turn out to be a tool for many of the considered integrals.
Best Answer
Indeed let $$ I(n,a)=\int_0^\infty\frac{dx}{\sqrt{x}(1+ax+x^2)(\sum_{k=0}^n(-x)^k)} $$ The change of variables $x\leftarrow 1/x$ yields $$ I(n,a)=\int_0^\infty\frac{(-1)^nx^{n+1}dx}{ \sqrt{x}(1+ax+x^2)(\sum_{k=0}^n(-x)^k)} $$ Thus $$ 2I(n,a)=\int_0^\infty\frac{1+x}{\sqrt{x}(1+ax+x^2)}dx= 2\int_0^\infty\frac{1+t^2}{ 1+at^2+t^4}dt $$ Or equivalently, setting $u=t-1/t$, $$ I(n,a)= \int_{-\infty}^\infty\frac{du}{ 2+a+u^2} =\frac{\pi}{\sqrt{2+a}}. $$