There is numerical evidence that
$$I=\int_0^1\frac{1}{\sqrt{1-x^2}}\arccos\left(\frac{3x^3-3x+4x^2\sqrt{2-x^2}}{5x^2-1}\right)\mathrm dx=\frac{3\pi^2}{8}-2\pi\arctan\frac12.$$
How can this be proved?
Wolfram does not find an antiderivative.
Here is the graph of $y=\frac{1}{\sqrt{1-x^2}}\arccos\left(\frac{3x^3-3x+4x^2\sqrt{2-x^2}}{5x^2-1}\right)$.
Based on recent experience with integrals involving inverse trigonometric functions (example), I guess a proof may involve a lot of substitutions. But I don't have any insight on how to approach this.
A search on approachzero did not turn up anything similar.
Context
If this can be proved, then we can answer the question Probability that the centroid of a triangle is inside its incircle, via @user170231's answer.
How I found the conjectured closed form
In my comment to @user170231's answer to the linked question, I note that the probability in that question is $1-\frac{12}{\pi^2}\int_0^{\pi/2}(y-x_+)dy$, which is equivalent to $1-\frac{12}{\pi^2}(\int_0^{\pi/2}ydy-I)\approx0.457993176$, where $I$ is the integral in this question.
Wolfram suggests that $0.457993176$ is $4-24\left(\frac{\arctan(1/2)}{\pi}\right)$, which implies that $I=\frac{3\pi^2}{8}-2\pi\arctan\frac12$.
Best Answer
The starting point strategy (optional)
Beside this integral, there were already two more related integrals posted here before, namely: $\int_0^\frac{\pi}{3}\arccos(2\sin^2 x-\cos x)dx$ and $\int_{\arccos(1/4)}^{\pi/2}\arccos\left(\cos x\left(2\sin^2x+\sqrt{1+4\sin^4x}\right)\right)$, so it might be confusing to see a different starting point for each of them. However, all share the same target in that we want to arrive at a point where we can utilize:
$$\int_a^b \frac{\arctan\left(f(x)\frac{\sqrt{g(x)}}{\sqrt{h(x)}} \right)}{p(x)\sqrt{g(x)}\sqrt{h(x)}}dx=\int_a^b \frac{1}{p(x)}\int_0^{f(x)} \frac{1}{h(x)+g(x)y^2}dydx$$
The most crucial thing here is to get rid of the square roots. So, those functions can mostly be anything containing polynomials, but from what I've observed, it's best to ensure that $g(x)$ and $h(x)$ have similar "weights" - e.g. don't keep $g(x)=a+bx^2$ and $h(x)=x(m+nx^2)$, either add the $x$ term to $g(x)$ or remove it from $h(x)$. It's also acceptable for one of those to simply be $1$. Additionally, it would be great for $p(x)$ not to contain other square roots. Moreover, there are two useful ways to get from the $\arccos$ to the $\arctan$ form:
$$\arccos x=\arctan\frac{\sqrt{1-x^2}}{x}\ \text{ and } \arccos x = 2\arctan\sqrt{\frac{1-x}{1+x}} $$
So we need the $\arccos$ argument to be acceptable before utilizing one of those two identities. Having $2\sin^2 x-\cos x$ is awesome since it has no square roots so we can try to convert directly to the $\arctan $ form (the second identity behaves nicer in this case). Note that having no square roots doesn't automatically ensure that we will obtain something nice afterwards (but it's a good indicator to try and convert the argument directly).
Having $\cos x\left(2\sin^2x+\sqrt{1+4\sin^4x}\right)$ is horrible due to the fourth power, since we can't substitute out $\sqrt{1+4\sin^4 x}$ like we would do for a simpler $\sqrt{a^2+b^2\sin^2 x}$. Here's where the trick with Vieta's relations comes in handy - as shown in the second linked post.
In our case we don't have fourth powers, only $\sqrt{2-x^2}$, so we don't need the above trick. The argument however is even messier and we still need to remove the square root, or at least to isolate it. Here there were 3 substitutions that came to my mind: $\sqrt{2-x^2}\to x$, $\frac{\sqrt{2-x^2}}{x}\to x$ and the combination of $x\to \frac{1}{x}$ followed by the Euler substitution $\sqrt{2x^2-1}-\sqrt 2 x\to x$. Luckily, the second one worked nicely (I reversed it to keep the same bounds).
Evaluation of the integral
$$\mathcal I = \int_0^1 \frac{\arccos\left(\frac{3x^3-3x+4x^2\sqrt{2-x^2}}{5x^2-1}\right) }{\sqrt{1-x^2}}dx\overset{\large \frac{x}{\sqrt{2-x^2}}\to x} = \sqrt 2\int_0^1 \frac{\arccos\left(\frac{3+x}{1+3x}\frac{\sqrt{2}\, x}{\sqrt{1+x^2}}\right)}{\sqrt{1-x^2}(1+x^2)}dx$$
$$=\sqrt 2 \int_0^1 \frac{\arctan\left(\frac{\sqrt{1-x^2}}{x}\small \color{red}{\frac{\sqrt{(1-x)(1+7x)}}{\sqrt 2\, (3+x)}}\right)}{\sqrt{1-x^2}(1+x^2)}dx = \sqrt 2\int_0^1 \frac{1}{1+x^2}\int_0^{\color{red}{f(x)}} \frac{x}{x^2+(1-x^2)y^2} dydx$$
$$\overset{y\to f(y)}=\sqrt 2 \int_0^1 \frac{1}{1+x^2} \int_x^1 \frac{-xf'(y)}{x^2+(1-x^2) f^2(y)} dydx = \int_0^1 \int_{x}^1 {\small \left(*\right)} dydx = \int_0^1 \int_0^y {\small \left(*\right)}dxdy$$
$$=\frac12 \int_0^1 \frac{(1-3y)\ln\left(\frac{(1-y)(1+7y)}{(1+3y)^2}\right)}{(1+y^2)\sqrt{(1-y)(1+7y)}}dy\overset{\large \sqrt{\frac{1-y}{1+7y}}\to y} = 2\int_0^1 \frac{1-5y^2}{1+6y^2+25y^4}\ln\left(\frac{1+y^2}{2y}\right)dy$$
$$\overset{\small IBP}=\frac12\int_0^1{{\frac{(1-y^2)\ln\left(\frac{1+2y+5y^2}{1-2y+5y^2}\right)}{y(1+y^2)}}}dy \overset{y\to \frac{1-y}{1+y}}=2\int_0^1 \frac{y\ln\left(\frac{2-2y+y^2}{1-2y+2y^2}\right)}{1-y^4}dy\overset{\small \bigstar}=\int_0^\infty\frac{y\ln\left(\frac{2-2y+y^2}{1-2y+2y^2}\right)}{1-y^4}dy$$
$$=\int_0^\infty\frac{y\ln\left(\frac{1+2y+2y^2}{2+2y+y^2}\right)+y\ln\left(\frac{4+y^4}{1+4y^4}\right)}{1-y^4}dy\overset{\small \bigstar \bigstar}=2\int_0^\infty \frac{y\ln y+y\ln\left(\frac{4+y^4}{1+4y^4}\right)}{1-y^4}dy=2\mathcal J+2\mathcal K$$
Note that above, $\small \bigstar$, stands for the combination of substituting $y \to \frac{1}{y}$ followed by averaging the resulting integral with the original one.
Also, to show $\small \bigstar \bigstar$, we can proceed as follows:
$$\int_0^\infty \frac{y\ln\left(\frac{4+y^4}{1+2y+2y^2}\right)}{1-y^4}dy\overset{y\to \frac{1}{y}}=\int_0^\infty \frac{y\ln\left(\frac{1+4y^4}{2+2y+y^2}\right)-y\ln(y^2)}{1-y^4}dy$$
$$\Rightarrow \int_0^\infty \frac{y\ln\left(\frac{1+2y+2y^2}{2+2y+y^2}\right)}{1-y^4}dy=\int_0^\infty \frac{y\ln(y^2)+y\ln\left(\frac{4+y^4}{1+4y^4}\right)}{1-y^4}dy$$
So to finalize, it remains to find $\mathcal J$ and $\mathcal K$. The first one is found here, after rewritting it as:
$$\mathcal J=\int_0^\infty \frac{y\ln y}{1-y^4}dy\overset{\bigstar}=2\int_0^1 \frac{y\ln y}{1-y^4}dy \overset{y^2 \to y}=\frac12\int_0^1 \frac{\ln y}{1-y^2}dy=-\frac{\pi^2}{16}$$
The second one can be calculated as follows:
$$\mathcal K=\int_0^\infty \frac{y\ln\left(\frac{4+y^4}{1+4y^4}\right)}{1-y^4}dy\overset{y^2\to y}=\frac12\int_0^\infty \frac{\ln\left(\frac{4+y^2}{1+4y^2}\right)}{1-y^2}dy\overset{y\to \frac{1-y}{1+y}}=\frac14\int_{-1}^1 \frac{\ln\left(\frac{1+\frac{6}{5}y+y^2}{1-\frac{6}{5}y+y^2}\right)}{y}dy$$
$$=\frac12 \int_{-1}^1 \int_{-\arcsin \frac{3}{5}}^{\arcsin \frac{3}{5}} \frac{\cos t}{(y+\sin t)^2+\cos^2 t} dtdy=\frac{\pi}{4} \int_{-\arcsin \frac{3}{5}}^{\arcsin \frac{3}{5}} dt = \frac{\pi}{2} \arcsin \frac{3}{5}$$
$$\Rightarrow \mathcal I = 2\mathcal J + 2\mathcal K= -\frac{\pi^2}{8}+\pi \arcsin \frac{3}{5} = \boxed{\frac{3\pi^2}{8}-2\pi \arctan \frac12}$$