Let $\mathcal{I}$ denote the value of the definite integral
$$\mathcal{I}:=\int_{0}^{1}\mathrm{d}u\,\sqrt{\frac{u^{2}-2-2\sqrt{u^{4}-u^{2}+1}}{4u^{6}-8u^{4}+8u^{2}-4}}\approx1.5436866339.$$
Note: the denominator of the radicand of the outer square root has the factorization
$$\begin{align}
4u^{6}-8u^{4}+8u^{2}-4
&=4t^{3}-8t^{2}+8t-4;~~~\small{\left[u^{2}=t\right]}\\
&=4\left(t^{3}-2t^{2}+2t-1\right)\\
&=4\left(t-1\right)\left(t^{2}-t+1\right).\\
\end{align}$$
Using the substitution $u^{2}=t$, the integral $\mathcal{I}$ can be rewritten as
$$\begin{align}
\mathcal{I}
&=\int_{0}^{1}\mathrm{d}u\,\sqrt{\frac{u^{2}-2-2\sqrt{u^{4}-u^{2}+1}}{4u^{6}-8u^{4}+8u^{2}-4}}\\
&=\int_{0}^{1}\mathrm{d}t\,\frac{1}{2\sqrt{t}}\sqrt{\frac{t-2-2\sqrt{t^{2}-t+1}}{4t^{3}-8t^{2}+8t-4}};~~~\small{\left[u=\sqrt{t}\right]}\\
&=\int_{0}^{1}\mathrm{d}t\,\frac{1}{2\sqrt{t}}\sqrt{\frac{t-2-2\sqrt{t^{2}-t+1}}{4\left(t-1\right)\left(t^{2}-t+1\right)}}\\
&=\frac14\int_{0}^{1}\mathrm{d}t\,\sqrt{\frac{2-t+2\sqrt{t^{2}-t+1}}{t\left(1-t\right)\left(t^{2}-t+1\right)}}.\\
\end{align}$$
Now the inner radical is just a square root of a quadratic function, which suggests that the integral might be further simplified using an appropriate Euler substitution.
Consider a substitution given implicitly by the relation
$$\sqrt{t^{2}-t+1}=t+x.$$
Solving for $t$, we obtain $t=\frac{1-x^{2}}{1+2x}$. The integral $\mathcal{I}$ is then transformed to
$$\begin{align}
\mathcal{I}
&=\frac14\int_{0}^{1}\mathrm{d}t\,\sqrt{\frac{2-t+2\sqrt{t^{2}-t+1}}{t\left(1-t\right)\left(t^{2}-t+1\right)}}\\
&=\frac14\int_{1}^{0}\mathrm{d}x\,\frac{(-2)\left(1+x+x^{2}\right)}{\left(1+2x\right)^{2}}\sqrt{\frac{3\left(1+x\right)}{\left(1-x\right)}\cdot\frac{\left(1+2x\right)}{x\left(2+x\right)}\cdot\frac{\left(1+2x\right)^{2}}{\left(1+x+x^{2}\right)^{2}}};~~~\small{\left[t=\frac{1-x^{2}}{1+2x}\right]}\\
&=\frac12\int_{0}^{1}\mathrm{d}x\,\sqrt{\frac{3\left(1+x\right)}{x\left(1-x\right)\left(2+x\right)\left(1+2x\right)}}\\
&=\frac{\sqrt{3}}{2}\int_{0}^{1}\mathrm{d}x\,\frac{\left(1+x\right)}{\sqrt{x\left(1-x\right)\left(1+x\right)\left(1+2x\right)\left(2+x\right)}}.\\
\end{align}$$
Next, look what happens when we transform the integral using the linear fractional transformation $x=\frac{1-y}{1+y}$:
$$\begin{align}
\mathcal{I}
&=\frac{\sqrt{3}}{2}\int_{0}^{1}\mathrm{d}x\,\frac{\left(1+x\right)}{\sqrt{x\left(1-x\right)\left(1+x\right)\left(1+2x\right)\left(2+x\right)}}\\
&=\frac{\sqrt{3}}{2}\int_{0}^{1}\mathrm{d}x\,\frac{\left(1+x\right)}{\sqrt{\left(1+x\right)^{6}\left(\frac{1-x}{1+x}\right)\left(\frac{x}{1+x}\right)\left(\frac{1}{1+x}\right)\left(\frac{1+2x}{1+x}\right)\left(\frac{2+x}{1+x}\right)}}\\
&=\frac{\sqrt{3}}{2}\int_{0}^{1}\mathrm{d}x\,\frac{\left(\frac{1}{1+x}\right)^{2}}{\sqrt{\left(\frac{1-x}{1+x}\right)\left(\frac{x}{1+x}\right)\left(\frac{1}{1+x}\right)\left(\frac{1+2x}{1+x}\right)\left(\frac{2+x}{1+x}\right)}}\\
&=\frac{\sqrt{3}}{2}\int_{1}^{0}\mathrm{d}y\,\frac{(-2)}{\left(1+y\right)^{2}}\cdot\frac{\left(\frac{1+y}{2}\right)^{2}}{\sqrt{y\left(\frac{1-y}{2}\right)\left(\frac{1+y}{2}\right)\left(\frac{3-y}{2}\right)\left(\frac{3+y}{2}\right)}};~~~\small{\left[x=\frac{1-y}{1+y}\right]}\\
&=\sqrt{3}\int_{0}^{1}\mathrm{d}y\,\frac{1}{\sqrt{y\left(1-y\right)\left(1+y\right)\left(3-y\right)\left(3+y\right)}}\\
&=\sqrt{3}\int_{0}^{1}\mathrm{d}y\,\frac{1}{\sqrt{y\left(1-y^{2}\right)\left(9-y^{2}\right)}}.\\
\end{align}$$
Recalling Euler's integral formula for the Gauss hypergeometric function: for real argument and parameters,
$$\int_{0}^{1}\mathrm{d}t\,\frac{t^{b-1}\left(1-t\right)^{c-b-1}}{\left(1-zt\right)^{a}}=\operatorname{B}{\left(b,c-b\right)}\,{_2F_1}{\left(a,b;c;z\right)};~~~\small{\left(a,b,c,z\right)\in\mathbb{R}^{4}\land0<b<c\land z<1},$$
(where $\operatorname{B}$ here denotes the usual beta function), we arrive at the following representation for $\mathcal{I}$ as a particular value of ${_2F_1}$:
$$\begin{align}
\mathcal{I}
&=\sqrt{3}\int_{0}^{1}\mathrm{d}y\,\frac{1}{\sqrt{y\left(1-y^{2}\right)\left(9-y^{2}\right)}}\\
&=\frac{1}{\sqrt{3}}\int_{0}^{1}\mathrm{d}y\,\frac{1}{\sqrt{y}\sqrt{1-y^{2}}\sqrt{1-\frac19y^{2}}}\\
&=\frac{1}{\sqrt{3}}\int_{0}^{1}\mathrm{d}t\,\frac{1}{2\sqrt{t}}\cdot\frac{1}{\sqrt[4]{t}\sqrt{1-t}\sqrt{1-\frac19t}};~~~\small{\left[y=\sqrt{t}\right]}\\
&=\frac{1}{2\sqrt{3}}\int_{0}^{1}\mathrm{d}t\,\frac{t^{b-1}\left(1-t\right)^{c-b-1}}{\left(1-zt\right)^{a}};~~~\small{\left[a:=\frac12,b:=\frac14,c:=\frac34,z:=\frac19\right]}\\
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(b,c-b\right)}\,{_2F_1}{\left(a,b;c;z\right)}\\
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(\frac14,\frac12\right)}\,{_2F_1}{\left(\frac12,\frac14;\frac34;\frac19\right)}.\\
\end{align}$$
Given $\left(a,b,z\right)\in\mathbb{R}_{>0}\times\mathbb{R}_{>0}\times\left(0,1\right)$, the Gauss hypergeometric function obeys the following two functional relations:
$${_2F_1}{\left(a,b;2b;z\right)}=\left(\frac{1+\sqrt{1-z}}{2}\right)^{-2a}\,{_2F_1}{\left(a,a-b+\frac12;b+\frac12;\left(\frac{1-\sqrt{1-z}}{1+\sqrt{1-z}}\right)^{2}\right)};~~~\small{b<a+\frac12},$$
and
$$\begin{align}
{_2F_1}{\left(a,b;\frac12;z\right)}
&=\frac{\Gamma{\left(a+\frac12\right)}\,\Gamma{\left(b+\frac12\right)}}{2\,\Gamma{\left(\frac12\right)}\,\Gamma{\left(a+b+\frac12\right)}}\bigg{[}{_2F_1}{\left(2a,2b;a+b+\frac12;\frac{1-\sqrt{z}}{2}\right)}\\
&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+{_2F_1}{\left(2a,2b;a+b+\frac12;\frac{1+\sqrt{z}}{2}\right)}\bigg{]}.\\
\end{align}$$
The following pair of identities are then immediate corollaries of the pair above by setting $b=a$: for $0<a\land0<z<1$,
$${_2F_1}{\left(a,\frac12;a+\frac12;z^{2}\right)}=\left(1+z\right)^{-2a}\,{_2F_1}{\left(a,a;2a;\frac{4z}{\left(1+z\right)^{2}}\right)},$$
and
$$\begin{align}
{_2F_1}{\left(a,a;\frac12;z\right)}
&=\frac{\left[\Gamma{\left(a+\frac12\right)}\right]^{2}}{2\,\Gamma{\left(\frac12\right)}\,\Gamma{\left(2a+\frac12\right)}}\bigg{[}{_2F_1}{\left(2a,2a;2a+\frac12;\frac{1-\sqrt{z}}{2}\right)}\\
&~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~+{_2F_1}{\left(2a,2a;2a+\frac12;\frac{1+\sqrt{z}}{2}\right)}\bigg{]}.\\
\end{align}$$
Continuing with our main evaluation of the integral $\mathcal{I}$, the quadratic transformations of ${_2F_1}$ given above allow us to reduce our integral to standard complete elliptic integrals:
$$\begin{align}
\mathcal{I}
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(\frac14,\frac12\right)}\,{_2F_1}{\left(\frac12,\frac14;\frac34;\frac19\right)}\\
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(\frac14,\frac12\right)}\,{_2F_1}{\left(\frac14,\frac12;\frac34;\frac19\right)}\\
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(\frac14,\frac12\right)}\,\frac{\sqrt{3}}{2}\,{_2F_1}{\left(\frac14,\frac14;\frac12;\frac34\right)}\\
&=\frac{1}{2\sqrt{3}}\operatorname{B}{\left(\frac14,\frac12\right)}\,\frac{\sqrt{3}}{2}\cdot\frac{\left[\Gamma{\left(\frac34\right)}\right]^{2}}{2\,\Gamma{\left(\frac12\right)}}\bigg{[}{_2F_1}{\left(\frac12,\frac12;1;\frac{2-\sqrt{3}}{4}\right)}+{_2F_1}{\left(\frac12,\frac12;1;\frac{2+\sqrt{3}}{4}\right)}\bigg{]}\\
&=\frac{\Gamma{\left(\frac14\right)}\,\Gamma{\left(\frac34\right)}}{8}\bigg{[}{_2F_1}{\left(\frac12,\frac12;1;\frac{2-\sqrt{3}}{4}\right)}+{_2F_1}{\left(\frac12,\frac12;1;\frac{2+\sqrt{3}}{4}\right)}\bigg{]}\\
&=\frac{\sqrt{2}\,\pi}{8}\bigg{[}{_2F_1}{\left(\frac12,\frac12;1;\frac{2-\sqrt{3}}{4}\right)}+{_2F_1}{\left(\frac12,\frac12;1;\frac{2+\sqrt{3}}{4}\right)}\bigg{]}\\
&=\frac{1}{2\sqrt{2}}\bigg{[}K{\left(\frac{\sqrt{2-\sqrt{3}}}{2}\right)}+K{\left(\frac{\sqrt{2+\sqrt{3}}}{2}\right)}\bigg{]}\\
&=\frac{1}{2\sqrt{2}}\bigg{[}K{\left(\frac{\sqrt{2-\sqrt{3}}}{2}\right)}+K^{\prime}{\left(\frac{\sqrt{2-\sqrt{3}}}{2}\right)}\bigg{]}\\
&=\frac{1}{2\sqrt{2}}\bigg{[}K{\left(\sin{\frac{\pi}{12}}\right)}+K^{\prime}{\left(\sin{\frac{\pi}{12}}\right)}\bigg{]}.\\
\end{align}$$
where here $K{(k)}$ is the complete elliptic integral of the first kind defined as a function of elliptic modulus $k$ by
$$K{(k)}:=\int_{0}^{1}\mathrm{d}x\,\frac{1}{\sqrt{(1-x^{2})(1-k^{2}x^{2})}};~~~\small{-1<k<1},$$
and $K^{\prime}{(k)}$ is the complementary complete elliptic integral of the first kind and is defined in terms of $K$ by
$$K^{\prime}{(k)}:=K{\left(\sqrt{1-k^{2}}\right)}.$$
We can complete our calculation by recognizing that the modulus $k=\sin{\frac{\pi}{12}}$ is in fact the third elliptic integral singular value, $k_{3}$.
Finally, we obtain:
$$\begin{align}
\mathcal{I}
&=\frac{1}{2\sqrt{2}}\bigg{[}K{\left(\sin{\frac{\pi}{12}}\right)}+K^{\prime}{\left(\sin{\frac{\pi}{12}}\right)}\bigg{]}\\
&=\frac{1}{2\sqrt{2}}\bigg{[}K{\left(k_{3}\right)}+K^{\prime}{\left(k_{3}\right)}\bigg{]}\\
&=\frac{1}{2\sqrt{2}}\bigg{[}1+\frac{K^{\prime}{\left(k_{3}\right)}}{K{\left(k_{3}\right)}}\bigg{]}K{\left(k_{3}\right)}\\
&=\frac{1+\sqrt{3}}{2\sqrt{2}}\,K{\left(k_{3}\right)}\\
&=\frac{1+\sqrt{3}}{2\sqrt{2}}\cdot\frac{\sqrt[4]{3}}{6}\operatorname{B}{\left(\frac12,\frac16\right)}\\
&=\frac{1+\sqrt{3}}{2^{5/2}\,3^{3/4}}\cdot\frac{\Gamma{\left(\frac12\right)}\,\Gamma{\left(\frac16\right)}}{\Gamma{\left(\frac23\right)}}\\
&=\frac{1+\sqrt{3}}{2^{11/6}\,3^{3/4}}\cdot\frac{\pi\,\Gamma{\left(\frac13\right)}}{\left[\Gamma{\left(\frac23\right)}\right]^{2}}\\
&=\frac{\sqrt{3+2\sqrt{3}}}{2^{10/3}}\cdot\frac{\left[\Gamma{\left(\frac13\right)}\right]^{3}}{\pi}.\blacksquare\\
\end{align}$$
Best Answer
Define $g_n(x)=\sqrt{1-\sqrt{1-\cdots\sqrt{1-\sqrt{x}}}}$, where there are $n$ radicals present. For instance $g_2(x)=\sqrt{1-\sqrt{x}}$ and $g_1(x) = \sqrt{x}$. Notice that for $n\ge 2$, $g_n(x)=g_{n-1}(1-\sqrt{x})$. Also define $$ I_{n,m} = \int_0^1 x^m \sin^{-1}(g_n(x))dx \overset{1-\sqrt{x}=t}{=} 2\int_0^1 (1-t)^{2m+1}\sin^{-1}(g_{n-1}(t))dt. $$ A recurrence relation is formed and we obtain $$ I_{n,m} = 2\sum_{r=0}^{2m+1}\binom{2m+1}{r}(-1)^rI_{n-1,r},$$ where we keep reducing until we reach terms of the form $I_{1,l}$ for $l\in \mathbb{N}$. This integral can be computed exactly, $$ I_{1,m} = \int_0^1 x^m \sin^{-1}(\sqrt{x})dx = 2\int_0^1 t^{2m+1}\sin^{-1}(t)dt = \frac{\pi}{2(m+1)}\left(1-\frac{\Gamma(m+3/2)}{\sqrt{\pi} (m+1)!} \right)$$ This also shows that the integral $I_{n,0}$ is always of the form $\frac{p}{q}\pi$ where $p$ and $q$ are rational numbers.
Addendum : An explanation of how I got the final integral. Using integration by parts and this result, $$\begin{align*}\int_0^1 t^{2m+1}\sin^{-1}(t)dt &= \left.\sin^{-1}(t) \frac{t^{2m+2}}{2m+2}\right|_{0}^1 -\frac{1}{2(m+1)}\int_0^1 \frac{t^{2m+2}}{\sqrt{1-t^2}}dt\\ &= \frac{\pi}{4(m+1)} - \frac{1}{2(m+1)} \cdot \sqrt{\pi}\frac{\Gamma(m+3/2)}{2\Gamma(m+2)} \\ &= \frac{\pi}{4(m+1)}\left(1-\frac{\Gamma(m+3/2)}{\sqrt{\pi} (m+1)!} \right).\end{align*}$$ For additional clarity on how the integrals are calculated, $$I_{1,0} = \frac{\pi}{2}\left[1-\frac{\Gamma\left(\frac{3}{2}\right)}{\sqrt{\pi}\cdot 1!}\right] = \frac{\pi}{2}\left[1-\frac{1}{2}\right] = \frac{\pi}{4}\\ I_{1,1} = \frac{\pi}{4}\left[1-\frac{\Gamma\left(\frac{5}{2}\right)}{\sqrt{\pi} \cdot 2!}\right] = \frac{\pi}{2}\left[1-\frac{3}{8}\right] = \frac{5\pi}{32} \\ I_{1,2}= \frac{\pi}{6}\left[1-\frac{\Gamma\left(\frac{7}{2}\right)}{\sqrt{\pi} \cdot 3!}\right] = \frac{\pi}{2}\left[1-\frac{5}{16}\right] = \frac{11\pi}{96} \\ I_{1,3}= \frac{\pi}{8}\left[1-\frac{\Gamma\left(\frac{9}{2}\right)}{\sqrt{\pi} \cdot 4!}\right] = \frac{\pi}{2}\left[1-\frac{35}{128}\right] = \frac{93\pi}{1024} $$ From these four integrals, $$I_{2,0} = 2I_{1,0}-2I_{1,1} = 2\cdot \frac{\pi}{4}-2\cdot \frac{5\pi}{32} = \frac{3}{16}\pi.$$ and $$I_{3,0} = 2I_{2,0}-2I_{2,1} = \frac{3\pi}{8}-4\left[I_{1,0}-3I_{1,1}+3I_{1,2}-I_{1,3}\right] \\ = \frac{3\pi}{8} - \pi +\frac{\pi}{2}+\frac{93}{256}\pi = \frac{61\pi}{256}$$ You have a lot of patience if you have computed $I_{4,0}$ by hand.
Addendum 2: Since you have asked for $I_{n,0}$, using the recurrence relation, $I_{n,0} = 2(I_{n-1,0}-I_{n-1,1}) = 2(2(I_{n-2,0}-I_{n-2,1})-2(I_{n-2,0}-3I_{n-2,1}+3I_{n-2,2}-I_{n-2,3}))$, thus, $$ \boxed{I_{n,0} = 4\big[ 2I_{n-2,1}-3I_{n-2,2}+I_{n-2,3}\big]}$$