A large portion of this answer is taken directly from this answer!
Enforce the substitution
$$y=\sqrt{1-2x^2}\qquad\qquad x^2=\frac {1-y^2}{2}\qquad\qquad\mathrm dx=-\frac {y}{\sqrt{2(1-y^2)}}\,\mathrm dy$$
The integral now becomes
$$\begin{align*}\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx & =\sqrt{2}\int\limits_{1/\sqrt{3}}^1\frac {y\left(\frac {\pi}2-\arctan y\right)}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\\ & =\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y\,\mathrm dy}{\sqrt{1-y^2}(3-y^2)}-\sqrt{2}\int\limits_{1/\sqrt3}^1\frac {y\arctan y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy\end{align*}$$
Splitting up the integrand, the first integral can be evaluated with the substitution $y\mapsto\sqrt{1-y^2}$, giving
$$\frac {\pi}{\sqrt{2}}\int\limits_{1/\sqrt{3}}^1\frac {y}{\sqrt{1-y^2}(3-y^2)}\,\mathrm dy=\frac {\pi^2}{12}$$
And enforcing the substitution $t=\sqrt{y}$ on the second integral, we have that
$$\int\limits_0^{1/\sqrt3}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac 1{\sqrt{2}}\int\limits_{1/3}^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt$$
The last integral is difficult, but using the same methodology and formulas as the answer I have linked above, we first rewrite the integral such that the lower limit is zero.
$$\int\limits_{1/3}^1\frac {\arctan\sqrt t}{(3-t)\sqrt{1-t}}\,\mathrm dt=\underbrace{\int\limits_0^1\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{J}-\underbrace{\int\limits_0^{1/3}\frac {\arctan\sqrt{t}}{(3-t)\sqrt{1-t}}\,\mathrm dt}_{K}$$
Next, we use the following formula to evaluate the right-hand side
$$\int\limits_0^x\frac {\arctan\sqrt t}{(a-t)\sqrt{b-t}}\,\mathrm dt=\frac 1{\sqrt{a-b}}S\left(\arctan\sqrt{\frac {b-x}{a-b}},\arctan\sqrt{\frac {b+1}{a-b}},\arctan\frac 1{\sqrt{a}}\right)$$
There are two important observations that will help us evaluate the two integrals, namely
- $S(0,\beta,\gamma)=\pi(\beta-\gamma)$
- When $\sin^2\alpha+\sin^2\gamma=\sin^2\beta$, then $S(\alpha,\beta,\gamma)=-\alpha^2+\beta^2-\gamma^2$
Substituting $a=3$ and $b=1$, then
$$\begin{align*}J & =\frac 1{\sqrt2}S\left(0,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{12\sqrt2}\\K & =\frac 1{\sqrt2}S\left(\frac {\pi}6,\frac {\pi}4,\frac {\pi}6\right)=\frac {\pi^2}{144\sqrt2}\end{align*}$$
Where I have used the first observation to evaluate $J$ and the second observation to evaluate $K$. Taking the difference $J-K$, then
$$\int\limits_{1/3}^1\frac {\arctan x}{(3-x)\sqrt{1-x}}\,\mathrm dx=\frac {11\pi^2}{144\sqrt2}$$
Putting everything together, we get that
$$\int\limits_0^{1/\sqrt{3}}\frac {\arctan\left(\frac 1{\sqrt{1-2x^2}}\right)}{1+x^2}\,\mathrm dx=\frac {\pi^2}{12}-\frac {11\pi^2}{288}\color{blue}{=\frac {13\pi^2}{288}}$$
Best Answer
I see that you only asked for advice and hints. Hence I will not give all the steps unless you request them. First, the solution $$ \int_0^1\frac{\text{arctan}(\lambda x)}{x\sqrt{1-x^2}}\text{d}x = \frac{\pi}{2}\text{arcsinh}(\lambda) $$
As suggested in the comments, the way to arrive at this expression is to "differentiate wrt to $\lambda$" also known as the Feynman's trick. The steps are the following.
First, define $f(x,\lambda) = \frac{\text{arctan}(\lambda x)}{x\sqrt{1-x^2}}$ and $F(\lambda) = \int_0^1\frac{\text{arctan}(\lambda x)}{x\sqrt{1-x^2}}\text{d}x = \int_0^1f(x,\lambda)\text{d}x$.
Second, note that $F(0) = 0$ since $\text{arctan}(0)=0$.
Now, compute $\frac{\partial f}{\partial\lambda}$. In this step, instead of the $\text{arctan}{\lambda x}$ term you will get a rational function.
Now, note that $\frac{dF}{d\lambda} = \int_0^1\frac{\partial f}{\partial\lambda}dx$ which is an integral that should be easier to compute. Just be careful for this particular case when substituting the limits of integration. You may need to compute the (one-sided) limit $x\to 1^-$ instead of just plugging in $x=1$.
Now that you have $\frac{dF}{d\lambda}$ just integrate $F(\lambda)=\int \frac{dF}{d\lambda} d\lambda + C$ and use the fact that $F(0)=0$ to compute the integration constant $C$.
After all these, you should be able to arrive at $(\pi/2)\text{arcsinh}(\lambda)$
If you want to spoil yourself, these are the sequence of computations I did in wolfram following this procedure to obtain the result. Click them only of you get stuck again: Step 3, Step 4: anti derivative, Step 4: limit, Step 5
Regarding the second integral you asked for, I think I can confirm that the procedure is the same. Some of the computations you made for the first integral will be useful.