Complete answer now!$$I=\int_0^\infty \frac{{\arctan(x^2)}}{x^4+x^2+1}dx {\overset{x=\frac1{t}}=}
\int_0^\infty \frac{\arctan\left(\frac{1}{t^2}\right)}{\frac{1}{t^4}+\frac{1}{t^2}+1}\frac{dt}{t^2}\overset{t=x}=\int_0^\infty \frac{{x^2\left(\frac{\pi}{2}-\arctan(x^2)\right)}}{x^4+x^2+1}dx $$
Now if we add the result with the original integral $I$ we get:
$$2I=\frac{\pi}{2}\int_0^\infty \frac{x^2}{x^4+x^2+1}dx+\int_0^\infty \frac{(1-x^2)\arctan(x^2)}{x^4+x^2+1}dx$$
$$\Rightarrow I = \frac12 \cdot \frac{\pi}{2}\cdot \frac{\pi}{2\sqrt 3}-\frac12 \int_0^\infty \frac{(x^2-1)\arctan(x^2)}{x^4+x^2+1}dx=\frac{\pi^2}{8\sqrt 3} -\frac12 J$$
Now in order to calculate $J\,$ we start by performing IBP:
$$J=\int_0^\infty \frac{(x^2-1)\arctan\left(x^2\right)}{x^4+x^2+1}dx =\int_0^\infty \arctan(x^2) \left(\frac12 \ln\left(\frac{x^2-x+1}{x^2+x+1}\right)\right)'dx=$$
$$=\underbrace{\frac{1}{2}\ln\left(\frac{x^2-x+1}{x^2+x+1}\right)\arctan(x^2)\bigg|_0^\infty}_{=0}+\int_0^\infty \frac{x}{1+x^4}\ln\left(\frac{x^2+x+1}{x^2-x+1}\right)dx$$
Substituting $x=\tan\left(t\right)$ and doing some simplifications yields:
$$J=\int_0^\frac{\pi}{2} \frac{2\sin (2t)}{3+\cos(4t)}\ln\left(\frac{2+\sin (2t)}{2-\sin (2t)}\right)dt\overset{2t=x}=\int_0^\pi \frac{\sin x}{3+\cos(2x)}\ln\left(\frac{2+\sin x}{2-\sin x}\right)dx=$$
$$=2\int_0^\frac{\pi}{2}\frac{\sin x}{3+\cos(2x)}\ln\left(\frac{2+\sin x}{2-\sin x}\right)dx=\int_0^\frac{\pi}{2}\frac{\cos x}{1+\sin^2 x}\ln\left(\frac{2+\cos x}{2-\cos x}\right)dx =$$
$$=\frac12\int_0^\pi \frac{\cos x}{1+\sin^2 x}\ln\left(\frac{2+\cos x}{2-\cos x}\right)dx\overset{\large{\tan\left(\frac{x}{2}\right)=t}}=\int_0^\infty \frac{1-t^2}{t^4+6t^2+1}\ln\left(\frac{\color{blue}{t^2+3}}{\color{red}{3t^2+1}}\right)dt$$
Splitting the integral into two parts followed by the substitution $\,\displaystyle{t=\frac{1}{x}}\,$ in the second part gives:
$$\int_0^\infty \frac{1-t^2}{t^4+6t^2+1}\ln(\color{red}{3t^2+1})dt =\int_0^\infty \frac{x^2-1}{x^4+6x^2+1}\ln\left(\color{red}{\frac{x^2+3}{x^2}}\right)dx$$
$$\Rightarrow J=\int_0^\infty \frac{1-x^2}{x^4+6x^2+1} \ln(\color{blue}{x^2+3})dx - \int_0^\infty \frac{1-x^2}{x^4+6x^2+1} {\left(\ln(\color{red}{x^2})-\ln(\color{red}{x^2+3})\right)}dx=$$
$$=2\int_0^\infty \frac{1-x^2}{x^4+6x^2+1}\ln\left(\color{purple}{\frac{x^2+3}{x}}\right)dx=2\int_0^\infty \left(\frac12\arctan\left(\frac{2x}{1+x^2}\right)\right)'\ln\left(\frac{x^2+3}{x}\right)dx=$$
$$=\underbrace{\arctan\left(\frac{2x}{1+x^2}\right)\ln\left(\frac{x^2+3}{x}\right)\bigg|_0^\infty}_{=0}-\int_0^\infty \arctan\left(\frac{2x}{1+x^2}\right)\left(\frac{2x}{x^2+3}-\frac{1}{x}\right)dx$$
$$\Rightarrow J=\int_0^\infty \arctan\left(\frac{2x}{1+x^2}\right)\frac{dx}{x}-\int_0^\infty \arctan\left(\frac{2x}{1+x^2}\right) \frac{2x}{x^2+3}dx=J_1-J_2$$
$$J_1=\int_0^\infty \arctan\left(\frac{2x}{1+x^2}\right)\frac{dx}{x}\overset{\large{x=\tan\left(\frac{t}{2}\right)}}=\int_0^\pi \frac{\arctan( \sin t)}{\sin t} dt\overset{t=x}=2\int_0^\frac{\pi}{2} \frac{\arctan( \sin x)}{\sin x} dx$$
In general, we have the following relation: $$\frac{\arctan x}{x}=\int_0^1 \frac{dy}{1+(xy)^2} \Rightarrow \color{red}{\frac{\arctan(\sin x)}{\sin x}=\int_0^1 \frac{dy}{1+(\sin^2 x )y^2}}$$
$$J_1 = 2\color{blue}{\int_{0}^{\frac{\pi}{2}}} \color{red}{\frac{\arctan\left(\sin x\right)}{\sin x}}\color{blue}{dx}=2\color{blue}{\int_0^\frac{\pi}{2}}\color{red}{\int_0^1 \frac{dy}{1+(\sin^2 x )y^2}}\color{blue}{dx}=2\color{red}{\int_0^1} \color{blue}{\int_0^\frac{\pi}{2}}\color{purple}{\frac{1}{1+(\sin^2 x )y^2}}\color{blue}{dx}\color{red}{dy}$$
$$=2\int_0^1 \frac{\arctan\left(\sqrt{1+y^2}\cdot\tan(x)\right) }{\sqrt{1+y^2}} \bigg|_0^\frac{\pi}{2}=\pi\int_0^1 \frac{dy}{\sqrt{1+y^2}}=\boxed{\pi\ln\left(1+\sqrt 2\right)}$$
In order to evaluate $J_2$ we return the integral before was integrated by parts.
$$J_2=2\int_0^\infty \arctan\left(\frac{2 x} {x^2 +1}\right)\frac{x}{x^2 +3}dx=2\int_0^{\infty}\frac{(x^2-1)\ln(x^2+3)}{x^4+6x^2+1} dx=$$
$$=(\sqrt 2+1)\int_0^{\infty} \frac{\ln(x^2+3)}{x^2+\left(\sqrt 2+1\right)^2} \ dx - (\sqrt 2-1)\int_0^{\infty} \frac{\ln(x^2+3)}{x^2+\left(\sqrt 2-1\right)^2} dx$$
Using the following identity that is valid for $a\ge 0, b>0$:$$\int_0^{\infty} \frac{\ln(x^2+a^2)}{x^2+b^2} \ dx = \frac{\pi}{b}\ln(a+b)$$ $$\Rightarrow J_2=\pi\ln\left(\frac{\sqrt{3}+\sqrt{2}+1}{\sqrt{3}+\sqrt{2}-1}\right)=\boxed{\frac{\pi} {2}\ln(2+\sqrt 3)}$$
So we found that:$$J=\boxed{\pi \ln(1+\sqrt 2) - \frac{\pi} {2} \ln(2+\sqrt 3)}\Rightarrow I= \large\boxed{\frac{\pi^2} {8 \sqrt 3}+\frac{\pi}{4}\ln(2+\sqrt 3)-\frac{\pi}{2} \ln(1+\sqrt 2)}$$
Another approach,
Perform integration by parts,
\begin{align*}
I&=\int_0^1 \frac{\arctan x}{x}\ln\left(\frac{1+x^2}{(1-x)^2}\right)\,dx\\
&=\Big[\ln (x) \ln\left(\frac{1+x^2}{(1-x)^2}\right)\arctan x\Big]_0^1 -\int_0^1 \frac{\ln x}{1+x^2}\ln\left(\frac{1+x^2}{(1-x)^2}\right)dx-\int_0^1 \frac{2(1+x)\ln (x)\arctan (x)}{(1-x)(1+x^2)}dx\\
&=-\int_0^1 \frac{\ln x}{1+x^2}\ln\left(\frac{1+x^2}{(1-x)^2}\right)dx-2\int_0^1 \frac{(1+x)\ln (x)\arctan (x)}{(1-x)(1+x^2)}dx\\
\end{align*}
For $x\in [0;1]$ define the function $R$ by,
\begin{align*}
R(x)=\int_0^x \frac{(1+t)\ln t}{(1-t)(1+t^2)}dt=\int_0^1 \frac{x(1+tx)\ln (tx)}{(1-tx)(1+t^2x^2)}dt\\
\end{align*}
Observe that,
\begin{align*}
R(1)=\int_0^1 \frac{t\ln t}{1+t}dt+\int_0^1 \frac{\ln t}{1-t}dt
\end{align*}
Perform integration by parts,
\begin{align*}
I&=-\int_0^1 \frac{\ln x}{1+x^2}\ln\left(\frac{1+x^2}{(1-x)^2}\right)dx-2\Big[R(x)\arctan x\Big]_0^1+2\int_0^1\frac{R(x)}{1+x^2}dx\\
&=-\int_0^1 \frac{\ln x}{1+x^2}\ln\left(\frac{1+x^2}{(1-x)^2}\right)dx-\frac{\pi}{2}R(1)+2\int_0^1 \int_0^1 \frac{x(1+tx)\ln (tx)}{(1-tx)(1+t^2x^2)(1+x^2)}dtdx\\
&=-\int_0^1 \frac{\ln x}{1+x^2}\ln\left(\frac{1+x^2}{(1-x)^2}\right)dx-\frac{\pi}{2}R(1)+\int_0^1 \ln x\left[\frac{1}{1+x^2}\ln\left(\frac{1+t^2x^2}{(1-tx)^2}\right)\right]_{t=0}^{t=1} dx+\\
&\int_0^1 \ln t\left[\frac{1}{1+t^2}\ln\left(\frac{1+x^2}{(1-tx)^2}\right)+\frac{2\arctan (tx)}{1-t^2}-\frac{2t\arctan x}{1+t^2}-\frac{2t\arctan x}{1-t^2}\right]_{x=0}^{x=1} dt\\
&=-\frac{\pi }{2}R(1)+\ln 2\int_0^1 \frac{\ln t}{1+t^2}dt-2\int_0^1 \frac{\ln (1-t)\ln t}{1+t^2}dt+2\int_0^1 \frac{\ln t\arctan t}{1-t^2}dt-\\
&\frac{\pi}{2} \int_0^1 \frac{t\ln t}{1+t^2}dt-\frac{\pi}{2} \int_0^1\frac{t\ln t}{1-t^2} dt\\
\end{align*}
For $x\in [0;1]$ define the function $S$ by,
\begin{align*}
S(x)=\int_0^x \frac{\ln t}{1-t^2}dt=\int_0^1 \frac{x\ln(tx)}{1-t^2x^2} dt
\end{align*}
Perform integration by parts,
\begin{align*}
\int_0^1 \frac{\ln x\arctan x}{1-x^2}dx&=\Big[S(x)\arctan x\Big]_0^1-\int_0^1 \frac{S(x)}{1+x^2}dx\\
&=\frac{\pi}{4}S(1)-\int_0^1 \int_0^1 \frac{x\ln(tx)} {(1-t^2x^2)(1+x^2)} dtdx\\
&=\frac{\pi}{4}S(1)-\frac{1}{2}\int_0^1 \left[ \frac{\ln x}{1+x^2}\ln\left(\frac{1+tx}{1-tx} \right)\right]_{t=0}^{t=1} dx-\\
&\frac{1}{2}\int_0^1 \left[ \frac{\ln t}{1+t^2}\ln\left(\frac{1+x^2}{1-t^2x^2} \right)\right]_{x=0}^{x=1}dt\\
&=\frac{\pi}{4}S(1)-\frac{\ln 2}{2}\int_0^1 \frac{\ln t}{1+t^2}dt+\int_0^1 \frac{\ln(1-x)\ln x}{1+x^2}dx
\end{align*}
Therefore,
\begin{align*}I&=\pi\int_0^1\frac{2t\ln t}{t^4-1} dt\end{align*}
Perform the change of variable $y=t^2$,
\begin{align*}I&=\frac{1}{2}\pi \int_0^1 \frac{\ln y}{y^2-1}dy\\
&=\frac{1}{2}\pi\times \frac{3}{4}\zeta(2)\\
&=\frac{\pi^3}{16}
\end{align*}
Best Answer
Here is one approach. As a warning, the final answer I find is not pretty.
Let $$I = \int_0^1 \frac{\tan^{-1} x}{x^2 -x - 1} \, dx.$$ Start by using a self-similar substitution of $$x = \frac{1 - u}{1 + u}, \,\, dx = -\frac{2}{(1 + u)^2} \, du.$$ So , after having reverted the dummy variable $u$ back to $x$, we have $$I = 2 \int_0^1 \frac{\tan^{-1} \left (\frac{1 - x}{1 + x} \right )}{x^2 - 4x - 1} \, dx.$$ Noting that for $0 < x < 1$ $$\tan^{-1} \left (\frac{1 - x}{1 + x} \right ) = \frac{\pi}{4} - \tan^{-1} x,$$ then \begin{align} I &= \frac{\pi}{2} \int_0^1 \frac{dx}{x^2 - 4x - 1} - 2 \int_0^1 \frac{\tan^{-1} x}{x^2 - 4x - 1} \, dx\\ &= -\frac{\pi}{2 \sqrt{5}} \coth^{-1} \left (\frac{3}{\sqrt{5}} \right ) - 2 J, \end{align} where $$J = \int_0^1 \frac{\tan^{-1} x}{x^2 - 4x - 1} \, dx.$$
To find $J$ we begin by noting that $\tan^{-1} x = \operatorname{Im} \ln (1 + ix)$. Thus $$J = \operatorname{Im} \int_0^1 \frac{\ln (1 + ix)}{x^2 - 4x - 1} \, dx.$$ Making a substitution of $t = 1 + ix$ we have $$J = - \operatorname{Re} \int_1^{1+i} \frac{\ln t}{(t - \alpha)(t - \beta)} \, dt,$$ where $\alpha = 1 + i(2 - \sqrt{5})$ and $\beta = 1 + i(2 + \sqrt{5})$. After performing a partial fraction decomposition we are left with $$J = \frac{1}{2 \sqrt{5}} \operatorname{Im} \left [\int_1^{1 + i} \frac{\ln t}{\alpha - t} \, dt - \int_1^{1 + i} \frac{\ln t}{\beta - t} \, dt \right ].$$ Now, as $$\int \frac{\ln x}{z - x} \, dx = - \ln \left (1 - \frac{x}{z} \right ) \ln x - \operatorname{Li}_2 \left (\frac{x}{z} \right ),$$ (for a proof of this result see the appendix below), one has $$J = \frac{1}{2 \sqrt{5}} \operatorname{Im} \left [\ln (1 + i) \ln \left (\frac{\alpha}{\beta} \cdot \frac{\beta - 1 - i}{\alpha - i - i} \right ) + \operatorname{Li}_2 \left (\frac{1}{\alpha} \right ) - \operatorname{Li}_2 \left (\frac{1}{\beta} \right ) + \operatorname{Li}_2 \left (\frac{1 + i}{\beta} \right ) - \operatorname{Li}_2 \left (\frac{1 + i}{\alpha} \right ) \right ]$$ or after performing a huge amount of algebra \begin{align} J &= \frac{\pi}{8 \sqrt{5}} \ln (\sqrt{5} - 1) + \frac{1}{2 \sqrt{5}} \operatorname{Im} \left [\operatorname{Li}_2 \left (\frac{1}{2} + \frac{1}{\sqrt{5}}+ \frac{i}{2 \sqrt{5}} \right ) - \operatorname{Li}_2 \left (\frac{1}{2} - \frac{1}{\sqrt{5}} - \frac{i}{2 \sqrt{5}} \right ) \right.\\ & \quad+ \left. \operatorname{Li}_2 \left (\frac{1}{2} -\frac{1}{2 \sqrt{5}} - i \left (\frac{3}{2 \sqrt{5}} - \frac{1}{2} \right ) \right ) - \operatorname{Li}_2 \left (\frac{1}{2} +\frac{1}{2 \sqrt{5}} + i \left (\frac{3}{2 \sqrt{5}} + \frac{1}{2} \right ) \right ) \right ]\\ &= \frac{\pi}{8 \sqrt{5}} \ln (\sqrt{5} - 1) + \frac{1}{2 \sqrt{5}} \operatorname{Im} \frak{w}, \end{align} where $\frak{w}$ is the term containing the four dilogarithms with complex arguments. Thus $$\int_0^1 \frac{\tan^{-1} x}{x^2 - x - 1} \, dx = -\frac{\pi}{4 \sqrt{5}} \left (\ln 2 + \sinh^{-1} (2) \right ) - \frac{1}{\sqrt{5}} \operatorname{Im} \frak{w}.$$ Note that as $\operatorname{Im} {\frak{w}} = -0.8363170651979\ldots$ we see that $I \approx -0.376513$.
Appendix
Proof of $$\int \frac{\ln x}{z - x} \, dx = - \ln \left (1 - \frac{x}{z} \right ) \ln x - \operatorname{Li}_2 \left (\frac{x}{z} \right ) + C$$
Setting $t = x/z, dt = dx/z$, we have \begin{align} \int \frac{\ln x}{z - x} \, dx &= \int \frac{\ln (zt)}{1 - t} \, dt\\ &= -\ln (1 - t) \ln (zt) + \int \frac{\ln (1 - t)}{t} \, dt \qquad \text{(by parts)}\\ &= -\ln (1 - t) \ln (zt) - \operatorname{Li}_2 (t) + C\\ &= - \ln \left (1 - \frac{x}{z} \right ) \ln x - \operatorname{Li}_2 \left (\frac{x}{z} \right ) + C, \end{align} as required to show.