I am unable to show
$$\int_0^{\frac1{\sqrt3}} \frac{\cot^{-1}\sqrt{2-x^2}}{1+x^2}dx=\frac{\pi^2}{30} $$
except with a few observations below.
1). Despite the appearance, I do not believe this integral is related to the Ahmed integral or any variations of known solution.
2). The integral is related to a more complex one posted here. However, the answer offered is non conventional, which I could not fully appreciate.
3). Numerically, I could establish that
$$\int_0^{\frac1{\sqrt3}} \frac{\cot^{-1}\sqrt{2-x^2}}{1+x^2}dx
=2 \int_0^{\frac1{\sqrt3}} \frac{\cot^{-1}\sqrt5 x}{1+x^2}dx
– \int_0^{\sqrt3} \frac{\cot^{-1}\sqrt5 x}{1+x^2}dx $$
where the RHS can be worked out with an elaborate procedure.
So, I would like to see a traditional solution without special functions, or, at least, establish the integral relationship above analytically.
Best Answer
Perform the substitution $t=\frac{1}{\sqrt{2-x^2}}$ to check that, $$ \tag{1}\int_{0}^{\frac{1}{\sqrt{3} } } \frac{\operatorname{arccot} \left ( \sqrt{2-x^2} \right ) }{ 1+x^2}\text{d}x= \int_{\frac{\sqrt{2}}{2} }^{\sqrt{\frac{3}{5}}} \frac{\arctan\left ( t \right ) }{\sqrt{2t^2-1}(3t^2-1) }\text{d}t. $$ The second integral can be done by evaluating $$ \tag{2}\int_{0}^{1}\int_{0}^{1}\frac{1}{(1+x^2)(1+y^2)\sqrt{3+x^2+y^2} }\text{d}x\text{d}y =\frac{\pi^2}{30}. $$ To prove $(2)$, use the identity $$ \int_{0}^{\infty} {e^{-t^2x^2}}\text{d}x =\frac{\sqrt{\pi} }{2\sqrt{t} }. $$ Then I can get (classical result $\displaystyle{\int_{0}^{1} \frac{e^{-t^2x^2}}{1+x^2}\text{d}x =\frac{\pi}{4}e^{t^2}\left ( 1-\operatorname{erf}(t)^2 \right )}$ is used) $$ \begin{align*} &\int_{0}^{1}\int_{0}^{1}\frac{1}{(1+x^2)(1+y^2)\sqrt{3+x^2+y^2} }\text{d}x\text{d}y\\ =& \frac{2}{\sqrt{\pi} } \int_{0}^{1} \int_{0}^{1} \frac{1}{(1+x^2)(1+y^2)} \left (\int_{0}^{\infty}e^{-t^2(3+x^2+y^2)}\text{d}t\right ) \text{d}x\text{d}y\\ \overset{{\scriptsize\text{Fubini}}}{=}& \frac{2}{\sqrt{\pi} } \int_{0}^{\infty}e^{-3t^2}\left ( \int_{0}^{1} \frac{e^{-t^2x^2}}{1+x^2}\text{d} x\right )^2\text{d}t\\ =&\frac{2}{\sqrt{\pi} }\cdot\frac{\pi^2}{16} \int_{0}^{\infty} e^{-x^2}\left(1-\operatorname{erf}(x)^2\right)^2\text{d}x\\ =&\frac{2}{\sqrt{\pi} }\cdot\frac{\pi^2}{16} \int_{0}^{\infty} e^{-x^2}\left(1-2\operatorname{erf}(x)^2+\operatorname{erf}(x)^4\right)\text{d}x\\ =&\frac{2}{\sqrt{\pi} }\cdot\frac{\pi^2}{16}\left(\int_{0}^{\infty}e^{-x^2}\text{d}x -2\int_{0}^{\infty}e^{-x^2}\operatorname{erf}(x)^2\text{d}x +\int_{0}^{\infty}e^{-x^2}\operatorname{erf}(x)^4\text{d}x\right)\\ =&\frac{\pi^2}{30}. \end{align*} $$ The last equality follows from their primitives $$ \frac{\mathrm{d} }{\mathrm{d} x} \left ( \frac{\sqrt{\pi}}{2} \operatorname{erf}(x)^{n+1} \right ) =(n+1)e^{-t^2}\operatorname{erf}(x)^{n}. $$ And $$ \begin{align*} &\int_{0}^{1}\int_{0}^{1}\frac{1}{(1+x^2)(1+y^2)\sqrt{3+x^2+y^2} }\text{d}x\text{d}y\\ =&\int_{0}^{1} \frac{\arctan\left ( \sqrt{\frac{2+x^2}{4+x^2} } \right ) }{ \left ( 1+x^2 \right )\sqrt{2+x^2} }\text{d}x \\ \overset{{\scriptsize t=\sqrt{\frac{2+x^2}{4+x^2} }}}{=}&\int_{\frac{\sqrt{2}}{2} }^{\sqrt{\frac{3}{5}}} \frac{\arctan\left ( t \right ) }{\sqrt{2t^2-1}(3t^2-1) }\text{d}t\\ =&\int_{0}^{\frac{1}{\sqrt{3} } } \frac{\operatorname{arccot} \left ( \sqrt{2-x^2} \right ) }{ 1+x^2}\text{d}x=\frac{\pi^2}{30}. \end{align*} $$