A large portion of this answer is taken directly from this answer!
Enforce the substitution
$$y=\sqrt{1-2x^2}\qquad\qquad x^2=\frac {1-y^2}{2}\qquad\qquad\mathrm dx=-\frac {y}{\sqrt{2(1-y^2)}}\,\mathrm dy$$
The integral now becomes
$$\begin{align*}\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx & =\sqrt{2}\int\limits_{1/\sqrt{3}}^1\frac {y\left(\frac {\pi}2-\arctan y\right)}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\\ & =\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y\,\mathrm dy}{\sqrt{1-y^2}(3-y^2)}-\sqrt{2}\int\limits_{1/\sqrt3}^1\frac {y\arctan y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\end{align*}$$
Splitting up the integrand, the first integral can be evaluated with the substitution $y\mapsto\sqrt{1-y^2}$, giving
$$\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy=\frac {\pi^2}{12}$$
And enforcing the substitution $t=\sqrt{y}$ on the second integral, we have that
$$\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac 1{\sqrt{2}}\int\limits_{1/3}^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt$$
The last integral is difficult, but using the same methodology and formulas as the answer I have linked above, we first rewrite the integral such that the lower limit is zero.
$$\int\limits_{1/3}^1\frac {\arctan\sqrt t}{(3-t)\sqrt{1-t}}\,\mathrm dt=\underbrace{\int\limits_0^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{J}-\underbrace{\int\limits_0^{1/3}\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{K}$$
Next, we use the following formula to evaluate the right-hand side
$$\int\limits_0^x\frac {\arctan\sqrt t}{(a-t)\sqrt{b-t}}\,\mathrm dt=\frac 1{\sqrt{a-b}}S\left(\arctan\sqrt{\frac {b-x}{a-b}},\arctan\sqrt{\frac {b+1}{a-b}},\arctan\frac 1{\sqrt{a}}\right)$$
There are two important observations that will help us evaluate the two integrals, namely
- $S(0,\beta,\gamma)=\pi(\beta-\gamma)$
- When $\sin^2\alpha+\sin^2\gamma=\sin^2\beta$, then $S(\alpha,\beta,\gamma)=-\alpha^2+\beta^2-\gamma^2$
Substituting $a=3$ and $b=1$, then
$$\begin{align*}J & =\frac 1{\sqrt2}S\left(0,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{12\sqrt2}\\K & =\frac 1{\sqrt2}S\left(\frac {\pi}6,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{144\sqrt2}\end{align*}$$
Where I have used the first observation to evaluate $J$ and the second observation to evaluate $K$. Taking the difference $J-K$, then
$$\int\limits_{1/3}^1\frac {\arctan x}{(3-x)\sqrt{1-x}}\,\mathrm dx=\frac {11\pi^2}{144\sqrt2}$$
Putting everything together, we get that
$$\int\limits_0^{1/\sqrt{3}}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac {11\pi^2}{288}\color{blue}{=\frac {13\pi^2}{288}}$$
Best Answer
Substituting $x=e^t$ gives the following: $$\begin{align}I_n&=\int_1^{\infty} \frac{1}{x-1}\frac{x^{1/n}-x^{-1/n}}{x^n}~dx\\&=\int_0^{\infty} \frac{e^{t/n}-e^{-t/n}}{e^{tn}(e^t-1)}\cdot e^t~dt\\&=\int_0^{\infty} \frac{e^{-t(n-1/n)}-e^{-t(n+1/n)}}{1-e^{-t}}~dt. \end{align}$$ Now compare with integral representation for the digamma function (due to Gauss) $$\psi(z)=\int_0^{\infty} \left(\frac{e^{-t}}{t}-\frac{e^{-zt}}{1-e^{-t}}\right)~dt$$ to obtain the same result as provided by @MariuszIwaniuk $$I_n=\psi\left(n+\frac{1}{n}\right)-\psi\left(n-\frac{1}{n}\right).$$ Since $n$ is an integer with $n>1$ (the integral doesn't converge for $n=1$), one can simplify the above result to a form without non-elementary functions. To do so, note that by using the recurrence for the digamma function $\psi(z+1)=\psi(z)+1/z$, one obtains inductively $$\psi\left(n+\frac{1}{n}\right)=\psi\left(\frac{1}{n}\right)+\sum_{k=0}^{n-1} \frac{1}{k+1/n},$$ $$\psi\left(n-\frac{1}{n}\right)=\psi\left(1-\frac{1}{n}\right)+n+\sum_{k=0}^{n-1} \frac{1}{k-1/n}.$$ Therefore, one has the following closed form $$\begin{align} I_n&=\psi\left(\frac{1}{n}\right)-\psi\left(1-\frac{1}{n}\right)-n+\sum_{k=0}^{n-1} \left(\frac{1}{k+1/n}-\frac{1}{k-1/n}\right)\\&=-\pi\cot(\pi/n)-n+\sum_{k=0}^{n-1} \frac{2n}{1-k^2 n^2}\\&=\left[n+\sum_{k=1}^{n-1} \frac{2n}{1-k^2 n^2}\right]-\pi\cot(\pi/n), \end{align}$$ where we have used the reflection formula $\psi(1-z)+\psi(z)=\pi\cot(\pi z)$.