$$\int_0^\frac{\pi}{3}\arccos(2\sin^2 x-\cos x)dx\overset{\large \tan\frac{x}{2}\to x}=2\int_0^\frac{1}{\sqrt 3}\frac{\arccos\left(\frac{-1+8x^2+x^4}{(1+x^2)^2}\right)}{1+x^2}dx$$
$$=4\int_0^\frac{1}{\sqrt 3} \frac{\arctan\left(\frac{1}{x}\sqrt{\frac{1-3x^2}{5+x^2}}\right)}{1+x^2}dx\overset{\large \frac{1}{x}\to\sqrt x}=2\int_3^\infty \frac{\arctan \left(\sqrt x\sqrt{\frac{x-3}{5x+1}}\right)}{\sqrt{x}(1+x)}dx$$
$$\overset{\large \frac{x-3}{5x+1}\to x}=8\int_0^\frac{1}{5} \frac{1}{(1-x)(1-5x)} \frac{\arctan\left(\sqrt x \sqrt{\frac{3+x}{1-5x}}\right)}{\sqrt{\frac{3+x}{1-5x}}} dx$$
$$=8\int_0^\frac{1}{5} \frac{1}{(1-x)(1-5x)} \left(\int_0^{\sqrt x} \frac{1}{1+\frac{3+x}{1-5x}y^2}dy\right)dx$$
$$=8\int_0^\frac{1}{\sqrt 5}\int_{y^2}^\frac{1}{5} \frac{1}{1-x} \frac{1}{(1+3y^2)-(5-y^2)x}dxdy=2\int_0^\frac{1}{\sqrt 5} \frac{\ln\left(\frac{1-y^2}{4y^2}\right)}{1-y^2}dy$$
$$\overset{y\to\frac{1-y}{1+y}}=\int_{\large\frac{1}{\phi^2}}^1 \frac{\ln\left(\large \frac{y}{(1-y)^2}\right)}{y}dy=2 \operatorname{Li}_2(1)-2\operatorname{Li}_2\left(\frac{1}{\phi^2}\right)-2\ln^2\phi=\boxed{\frac{\pi^2}{5}}$$
$$\operatorname{Li}_2(1)=\frac{\pi^2}{6},\quad \operatorname{Li}_2\left(\frac{1}{\phi^2}\right) = \frac{\pi^2}{15}-\ln^2\phi,\quad \phi=\frac{1+\sqrt 5}{2}$$
Some algebra to arrive at a manageable integral
Notice that the $\arccos $ argument is the "$+$" solution of a quadratic equation:
$$\cos x\left(2\sin^2x\pm\sqrt{1+4\sin^4x}\right)=\frac{2\sin x \sin(2x)\pm\sqrt{4\sin^2 x\sin^2(2x)+4\cos^2 x}}{2}$$
More exactly, with $b=-2\sin x\sin(2x)$ and $c=-\cos^2x$ in $y_{\pm}=\frac{-b\pm\sqrt{b^2-4c}}{2}$.
We also have $\arccos y_-\pm\arccos y_+=\arccos\left(y_-y_+\mp\sqrt{1-(y_-+y_+)^2 +2y_-y_+ +y_-^2y_+^2}\right)$, and since Vieta's relations allows us to utilize $y_-+y_+=2\sin x\sin(2x)$ and $\ y_-y_+=-\cos^2 x$, it makes sense to consider the following integrals:
$$I_{\pm}=\int_{\arccos(1/4)}^{\pi/2}\arccos\left(\cos x\left(2\sin^2x\pm\sqrt{1+4\sin^4x}\right)\right) dx$$
The original integral can then be rewritten as $I_+=\frac12\left((I_- + I_+)-(I_- - I_+)\right)$, where:
$$I_- \pm I_+=\int_{\arccos(1/4)}^{\pi/2}\arccos\left(-\cos^2 x\mp\sin^2 x\sqrt{1-16\cos^2 x}\right)dx$$
$$\overset{\sqrt{1-16\cos^2 x}\to x}=\int_0^1 \frac{x\arccos \left(-\frac{1-x^2}{16}\mp x\frac{15+x^2}{16}\right)}{\sqrt{1-x^2}{\sqrt{15+x^2}}}dx$$
Moreover, substituting $x\to -x$ in $I_- - I_+$, gives us:
$$I_+=\frac12\int_{-1}^1 \frac{x\arccos \left(-\frac{1-x^2}{16}- x\frac{15+x^2}{16}\right)}{\sqrt{1-x^2}{\sqrt{15+x^2}}}dx=\int_{-1}^1 \frac{x\operatorname{arctan}\sqrt{\frac{1-x^2}{15+x^2} \left(1+\frac{16}{(1-x)^2}\right)}}{\sqrt{1-x^2}{\sqrt{15+x^2}}}dx$$
In the last step it was utilized that $\arccos x =2\arctan \sqrt{\frac{1-x}{1+x}}$.
Evaluation of the integral
$$\int_{-1}^1 \frac{x\operatorname{arctan}\sqrt{\frac{1-x^2}{15+x^2} \left(1+\frac{16}{(1-x)^2}\right)}}{\sqrt{1-x^2}{\sqrt{15+x^2}}}dx=\int_{-1}^1 \int_0^{\sqrt{1+\frac{16}{(1-x)^2}}} \frac{x}{(15+x^2)+(1-x^2) y^2}dy dx$$
$$\overset{(*)}=16\int_{-1}^1 \int_0^{x} \frac{x}{(15+x^2)+(1-x^2) \left(1+\frac{16}{(1-y)^2}\right)}\frac{1}{(1-y)^2\sqrt{16+(1-y)^2}} dy dx$$
$$=16\int_{-1}^1 \frac{1}{(1-y)^2\sqrt{16+(1-y)^2}}\int_y^1 \frac{x}{(15+x^2)+\left(1+\frac{16}{(1-y)^2}\right)(1-x^2)} dx dy$$
$$=-\frac12\int_{-1}^1 \frac{\ln\left(\frac{1-y}{2}\right)}{\sqrt{16+(1-y)^2}}dy\overset{\frac{1-y}{2}\to y}=-\frac12 \int_0^1 \frac{\ln y}{\sqrt{4+y^2}}dy$$
$$\overset{y \to \frac{1-y^2}{y}}=\frac12 \int_{\large \frac{1}{\phi}}^1 \frac{\ln\left(\frac{y}{1-y^2}\right)}{y}dy =\frac14\operatorname{Li}_2(1) - \frac14\operatorname{Li}_2\left(\frac{1}{\phi^2}\right) - \frac{1}{4}\ln^2\phi \overset{(**)}=\, \boxed{\frac{\pi^2}{40}}$$
In $(**)$, the following dilogarithm values were employed:
$$\operatorname{Li}_2(1)=\frac{\pi^2}{6},\quad \operatorname{Li}_2\left(\frac{1}{\phi^2}\right) = \frac{\pi^2}{15}-\ln^2\phi,\quad \phi=\frac{1+\sqrt 5}{2}$$
Also, the $(*)$ step is not necessary, however I didn't know how to change the order of integration directly, so I had to make the substitution $y^2\to 1+\frac{16}{(1-y)^2}$ after noticing that the $x$-integral is odd which allows to write $\int_{-1}^1 \int_0^{f(x)}dydx = -\int_{-1}^1 \int_{f(x)}^\infty dydx$. Finally, $-\int_{-1}^1 \int_x^1 dydx$ was also rewritten as $\int_{-1}^1 \int_0^x dydx$ in order to change the order of integration easier.
Best Answer
Applying $\arcsin x=\frac12\arccos(1-2x^2)$ followed by $\arccos x=\frac12\arccos(2x^2-1)$ yields: $$ \arcsin\left(\frac14\sqrt{8-2\sqrt{10-2\sqrt{17-8\cos x}}}\right)=\frac14\arccos\left(\frac14-\frac14\sqrt{17-8\cos x}\right)=\frac14f(x)$$
Based on the graph from below, we can rewrite the red part, $\int_0^\pi f(x)dx$, as the total square area $(\pi^2)$ minus the blue part: $\int_{2\pi/3}^\pi f^{-1}(y)dy$, where $f^{-1}(y)=\arccos\left(2\sin^2y+\cos y\right)$.
$$\Rightarrow \frac14\int_0^\pi \arccos\left(\frac14-\frac14\sqrt{17-8\cos x}\right)dx=\frac{\pi^2}{4}-\frac14\int_\frac{2\pi}{3}^\pi\arccos\left(2\sin^2y+\cos y\right)dy$$ $$\overset{y\to \pi-y}=\frac{\pi^2}{4}-\frac14\int_0^\frac{\pi}{3}\arccos\left(2\sin^2y-\cos y\right)dy=\frac{\pi^2}{4}-\frac{\pi^2}{20}=\frac{\pi^2}{5}$$
The last integral can be found here.