Someone on Youtube posted a video solving this integral.
I can't find on Math.stack.exchange this integral using search engine https://approach0.xyz
It is related to $\displaystyle \int_0^\infty \frac{\arctan x\ln x}{1+x^2}dx$
Following is a solution that is not requiring the use of series:
\begin{align}J&=\int_0^1 \frac{\arcsin x\arccos x}{x}dx\\
&\overset{\text{IBP}}=\underbrace{\Big[\arcsin x\arccos x\ln x\Big]_0^1}_{=0}-\underbrace{\int_0^1 \frac{\arccos x\ln x}{\sqrt{1-x^2}}dx}_{x=\cos t }+\underbrace{\int_0^1 \frac{\arcsin x\ln x}{\sqrt{1-x^2}}dx}_{x=\sin t}\\
&=\int_0^{\frac{\pi}{2}} t\ln(\tan t)dt\\
&\overset{u=\tan t}=\int_0^\infty \frac{\arctan u\ln u}{1+u^2}du\\
&\overset{\text{IBP}}=\underbrace{\left[\arctan u\left(\int_0^u \frac{\ln t}{1+t^2}dt\right)\right]_0^\infty}_{=0}-\int_0^\infty \frac{1}{1+u^2}\left(\underbrace{\int_0^u \frac{\ln t}{1+t^2}dt}_{y(t)=\frac{t}{u}}\right)du\\
&=-\int_0^\infty \left(\int_0^1 \frac{u\ln(uy)}{(1+u^2)(1+u^2y^2)}dy\right)du\\
&=-\int_0^\infty \left(\int_0^1 \frac{u\ln u}{(1+u^2)(1+u^2y^2)}dy\right)du-\int_0^1 \left(\int_0^\infty \frac{u\ln y}{(1+u^2)(1+u^2y^2)}du\right)dy\\
&=-\int_0^\infty \left[\frac{\arctan(uy)}{1+u^2}\right]_{y=0}^{y=1}\ln udu-\frac{1}{2}\int_0^1 \left[\frac{\ln\left(\frac{1+u^2}{1+u^2y^2}\right)}{1-y^2}\right]_{u=0}^{u=\infty}\ln ydy\\
&=-J+\int_0^1 \frac{\ln^2 y}{1-y^2}dy\\
&=\frac{1}{2}\int_0^1 \frac{\ln^2 y}{1-y}dy-\frac{1}{2}\underbrace{\int_0^1 \frac{y\ln^2 y}{1-y^2}dy}_{z=y^2}\\
&=\frac{7}{16}\int_0^1 \frac{\ln^2 y}{1-y}dy\\
&=\frac{7}{16}\times 2\zeta(3)=\boxed{\frac{7}{8}\zeta(3)}
\end{align}
NB:I assume $\displaystyle \int_0^1 \frac{\ln^2 y}{1-y}dy=2\zeta(3)$
Feel free to post your solution.
Best Answer
Let $J(a)=\int_0^\infty \frac{\tan^{-1}(ax)\ln x}{1+x^2}$ \begin{align} J’(a)=& \int_0^\infty \frac{x\ln x}{(1+x^2)(1+a^2x^2)} \overset{x\to \frac1{ax}}{dx}\\ = & -\frac1{2}\int_0^\infty \frac{x\ln a}{(1+x^2)(1+{a^2}x^2)} {dx} = \frac{\ln^2 a}{2(1-a^2)} \end{align} which leads to \begin{align}\int_0^\infty \frac{\arctan x\ln x}{1+x^2}dx =\int_0^1J’(a)da= \frac12 \int_0^1 \frac{\ln^2 a}{1-a^2}da =\frac78\zeta(3) \end{align} Besides, the integral is also related to $$\int_0^1 \frac{\sin^{-1} x\cos^{-1} x}{x}dx =\frac14\int_0^\infty \frac{x^2}{\sinh x}dx =\frac18\int_0^\pi\frac{x(\pi-x)}{\sin x}dx=\frac78\zeta(3)$$