Update: Finally, a complete solution. Sorry it took so long.
Split the integral up into 3.
\begin{align}
I
&=-\int^{\sqrt{2}}_1\frac{\log{x}}{x}dx+\int^{\sqrt{2}}_1\frac{\log{((x^2-1)^2+1)}}{x}dx-\int^{\sqrt{2}}_1\frac{\log{((x-1)^2+1)}}{x}dx\\
&=-\frac{1}{8}(\log{2})^2+\frac{1}{2}\int^1_0\frac{\log(1+x^2)}{1+x}dx-\int^{\sqrt{2}-1}_0\frac{\log(1+x^2)}{1+x}dx
\end{align}
The second integral is rather easy to evaluate.
\begin{align}
\frac{1}{2}\int^1_0\frac{\log(1+x^2)}{1+x}dx
&=\frac{1}{2}\int^1_0\int^1_0\frac{x^2}{(1+x)(1+ax^2)}dx \ da\tag1\\
&=\frac{1}{2}\int^1_0\frac{1}{1+a}\int^1_0\frac{1}{1+x}+\frac{x-1}{1+ax^2}dx \ da\\
&=\frac{1}{2}\int^1_0\frac{\log{2}}{1+a}+\frac{\log(1+a)}{2a(1+a)}-\underbrace{\frac{\arctan(\sqrt{a})}{\sqrt{a}(1+a)}}_{\text{Let} \ y=\arctan{\sqrt{a}}}da\\
&=\frac{1}{2}\left[(\log{2})^2+\frac{1}{2}\underbrace{\int^1_0\frac{\log(1+a)}{a}da}_{-\operatorname{Li}_2(-1)=\frac{\pi^2}{12}}-\frac{1}{2}\underbrace{\int^1_0\frac{\log(1+a)}{1+a}da}_{\frac{1}{2}(\log{2})^2}-\frac{\pi^2}{16}\right]\\
&=\frac{3}{8}(\log{2})^2-\frac{\pi^2}{96}
\end{align}
The third integral can be evaluated using dilogarithms.
\begin{align}
\int^{\sqrt{2}-1}_0\frac{\log(1+x^2)}{1+x}dx
&=\sum_{r=\pm i}\int^{\sqrt{2}-1}_0\frac{\log(r+x)}{1+x}dx\tag2\\
&=-\sum_{r=\pm i}\int^{\frac{\lambda}{\sqrt{2}}}_{\lambda}\log\left(r-1+\frac{\lambda}{y}\right)\frac{dy}{y}\tag3\\
&=-\sum_{r=\pm i}\int^{\frac{r-1}{\sqrt{2}}}_{r-1}\frac{\log(1+y)}{y}-\frac{1}{y}\log\left(\frac{y}{r-1}\right)dy\tag4\\
&=\frac{1}{4}(\log{2})^2+\sum_{r=\pm i}\mathrm{Li}_2\left(\frac{1-r}{\sqrt{2}}\right)-\mathrm{Li}_2(1-r)\tag5\\
&=\frac{1}{4}(\log{2})^2+\mathrm{Li}_2(e^{i\pi/4})+\mathrm{Li}_2(e^{-i\pi/4})-\mathrm{Li}_2(\sqrt{2}e^{i\pi/4})-\mathrm{Li}_2(\sqrt{2}e^{-i\pi/4})\\
&=\frac{1}{4}(\log{2})^2-\frac{\pi^2}{96}\tag6\\
\end{align}
It follows that
$$I=-\frac{1}{8}(\log{2})^2+\frac{3}{8}(\log{2})^2-\frac{\pi^2}{96}-\frac{1}{4}(\log{2})^2+\frac{\pi^2}{96}=0$$
Explanation
$(1)$: Differentiate under the integral sign
$(2)$: Factorise $1+x^2$, let $r=\pm i$
$(3)$: Let $\displaystyle y=\frac{\lambda}{1+x}$
$(4)$: Let $\lambda=r-1$
$(5)$: Recognise that $\displaystyle\int\frac{\ln(1+y)}{y}dy=-\mathrm{Li}_2(-y)+C$ and $\displaystyle\int\frac{\ln(ay)}{y}dy=\frac{1}{2}\ln^2(ay)+C$
$(6)$: Use the identities here
$\newcommand{\+}{^{\dagger}}
\newcommand{\angles}[1]{\left\langle\, #1 \,\right\rangle}
\newcommand{\braces}[1]{\left\lbrace\, #1 \,\right\rbrace}
\newcommand{\bracks}[1]{\left\lbrack\, #1 \,\right\rbrack}
\newcommand{\ceil}[1]{\,\left\lceil\, #1 \,\right\rceil\,}
\newcommand{\dd}{{\rm d}}
\newcommand{\down}{\downarrow}
\newcommand{\ds}[1]{\displaystyle{#1}}
\newcommand{\expo}[1]{\,{\rm e}^{#1}\,}
\newcommand{\fermi}{\,{\rm f}}
\newcommand{\floor}[1]{\,\left\lfloor #1 \right\rfloor\,}
\newcommand{\half}{{1 \over 2}}
\newcommand{\ic}{{\rm i}}
\newcommand{\iff}{\Longleftrightarrow}
\newcommand{\imp}{\Longrightarrow}
\newcommand{\isdiv}{\,\left.\right\vert\,}
\newcommand{\ket}[1]{\left\vert #1\right\rangle}
\newcommand{\ol}[1]{\overline{#1}}
\newcommand{\pars}[1]{\left(\, #1 \,\right)}
\newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
\newcommand{\pp}{{\cal P}}
\newcommand{\root}[2][]{\,\sqrt[#1]{\vphantom{\large A}\,#2\,}\,}
\newcommand{\sech}{\,{\rm sech}}
\newcommand{\sgn}{\,{\rm sgn}}
\newcommand{\totald}[3][]{\frac{{\rm d}^{#1} #2}{{\rm d} #3^{#1}}}
\newcommand{\ul}[1]{\underline{#1}}
\newcommand{\verts}[1]{\left\vert\, #1 \,\right\vert}
\newcommand{\wt}[1]{\widetilde{#1}}$
With $\ds{r \equiv {1 + \root{3}\ic \over 2} = \expo{\pi\ic/3}}$
\begin{align}&\color{#c00000}{
\int_{0}^{1}{\ln\pars{\ln\pars{1/x}} \over x^{2} - x + 1}\,\dd x}
=\int_{0}^{1}{\ln\pars{\ln\pars{1/x}} \over \pars{x - r}\pars{x - r^{*}}}\,\dd x
\\[3mm]&=\int_{0}^{1}\ln\pars{\ln\pars{1/x}}
\pars{{1 \over x - r} - {1 \over x - r^{*}}}\,{1 \over r - r^{*}}\,\dd x
\\[3mm] & =
{1 \over \Im\pars{r}}\,\Im\int_{0}^{1}
{\ln\pars{\ln\pars{1/x}} \over x - r}\,\dd x
\end{align}
With $\ds{x \equiv \expo{-t}}$:
\begin{align}&\color{#c00000}{
\int_{0}^{1}{\ln\pars{\ln\pars{1/x}} \over x^{2} - x + 1}\,\dd x}
={2\root{3} \over 3}\Im\int_{\infty}^{0}{\ln\pars{t} \over \expo{-t} - r}
\,\pars{-\expo{-t}\,\dd t}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\bracks{{1 \over r}\int_{0}^{\infty}
{\ln\pars{t}\expo{-t} \over 1 - \expo{-t}/r}\,\dd t}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\bracks{{1 \over r}
\sum_{n = 1}^{\infty}{1 \over r^{n - 1}}\int_{0}^{\infty}
\ln\pars{t}\expo{-nt}\,\dd t}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\lim_{\mu\ \to\ 0}\partiald{}{\mu}\bracks{
\sum_{n = 1}^{\infty}r^{-n}\int_{0}^{\infty}t^{\mu}\expo{-nt}\,\dd t}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\lim_{\mu\ \to\ 0}\partiald{}{\mu}\bracks{
\sum_{n = 1}^{\infty}{r^{-n} \over n^{\mu + 1}}\Gamma\pars{\mu + 1}}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\lim_{\mu\ \to\ 0}\partiald{}{\mu}\bracks{
\Gamma\pars{\mu + 1}{\rm Li}_{\mu + 1}\pars{r^{*}}}
\\[3mm]&=-\,{2\root{3} \over 3}\Im\lim_{\mu\ \to\ 0}\partiald{}{\mu}\bracks{
\Gamma\pars{\mu + 1}{\rm Li}_{\mu + 1}\pars{\expo{-\pi\ic/3}}}
\end{align}
$\ds{{\rm Li}_{1}\pars{z} = -\ln\pars{1 - z}}$. Derivatives of the
PolyLogarithm, respect of the order, can be evaluated from its integral representation.
Also, see Hurwitz Zeta Function.
Best Answer
Let $I_n(a,b)$ be the desired integral. Note that $I_n(a,b)=I_n(b,a)$, and $I_n(a,b)=I_n(-a,-b)$. So, we may suppose that $|b|< a$ Note that $$\eqalign{ \frac{a^2-b^2}{a^2-2ab\cos x+b^2}&=\frac{a}{a-e^{ix}b}+\frac{be^{-ix}}{a-e^{-ix}b}\cr &=\sum_{n=0}^\infty \left(\frac{b}{a}\right)^ne^{inx}+\frac{be^{-ix}}{a}\sum_{n=0}^\infty \left(\frac{b}{a}\right)^ne^{-inx}\cr &=1+\sum_{n=1}^\infty \left(\frac{b}{a}\right)^ne^{inx}+ \sum_{n=1}^{\infty} \left(\frac{b}{a}\right)^{n}e^{-inx}\cr &=1+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\cos(n x) } $$ It follows, using the uniform convergence of the series on $[0,\pi]$, that $$ \int_0^\pi\frac{(a^2-b^2)\cos(mx)}{a^2-2ab\cos x+b^2}dx =\int_0^\pi\cos(mx)dx+2\sum_{n=1}^\infty \left(\frac{b}{a}\right)^n\int_0^\pi\cos(n x)\cos(mx)dx $$ But $\int_0^\pi\cos(n x)\cos(mx)dx=0$ if $n\ne m$, and $\int_0^\pi\cos^2(n x)dx=\pi/2$ if $n\ne0$. So $$\eqalign{I_m(a,b)= \int_0^\pi\frac{\cos(mx)}{a^2-2ab\cos x+b^2}dx &=\left\{\matrix{\frac{\pi}{a^2-b^2}&\hbox{if}&m=0\cr \frac{\pi}{a^2-b^2}\left(\frac{b}{a}\right)^m&\hbox{if}&m\ne0 } \right.} $$ which is the desired formula for $|b|<a$.