I will transform the integral via a substitution, break it up into two pieces and recombine, perform an integration by parts, and perform another substitution to get an integral to which I know a closed form exists. From there, I use a method I know to attack the integral, but in an unusual way because of the 8th degree polynomial in the denominator of the integrand.
First sub $t=(1-x)/(1+x)$, $dt=-2/(1+x)^2 dx$ to get
$$2 \int_0^{\infty} dt \frac{t^{-1/2}}{1-t^2} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )} $$
Now use the symmetry from the map $t \mapsto 1/t$. Break the integral up into two as follows:
\begin{align}
& 2 \int_0^{1} dt \frac{t^{-1/2}}{1-t^2} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )} + 2 \int_1^{\infty} dt \frac{t^{-1/2}}{1-t^2} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )} \\
&= 2 \int_0^{1} dt \frac{t^{-1/2}}{1-t^2} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )} + 2 \int_0^{1} dt \frac{t^{1/2}}{1-t^2} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )} \\
&= 2 \int_0^{1} dt \frac{t^{-1/2}}{1-t} \log{\left (\frac{5-2 t+t^2}{1-2 t +5 t^2} \right )}
\end{align}
Sub $t=u^2$ to get
$$4 \int_0^{1} \frac{du}{1-u^2} \log{\left (\frac{5-2 u^2+u^4}{1-2 u^2 +5 u^4} \right )}$$
Integrate by parts:
$$\left [2 \log{\left (\frac{1+u}{1-u} \right )} \log{\left (\frac{5-2 u^2+u^4}{1-2 u^2 +5 u^4} \right )}\right ]_0^1 \\- 32 \int_0^1 du \frac{\left(u^5-6 u^3+u\right)}{\left(u^4-2 u^2+5\right) \left(5 u^4-2 u^2+1\right)} \log{\left (\frac{1+u}{1-u} \right )}$$
One last sub: $u=(v-1)/(v+1)$ $du=2/(v+1)^2 dv$, and finally get
$$8 \int_0^{\infty} dv \frac{(v^2-1)(v^4-6 v^2+1)}{v^8+4 v^6+70v^4+4 v^2+1} \log{v}$$
With this form, we may finally conclude that a closed form exists and apply the residue theorem to obtain it. To wit, consider the following contour integral:
$$\oint_C dz \frac{8 (z^2-1)(z^4-6 z^2+1)}{z^8+4 z^6+70z^4+4 z^2+1} \log^2{z}$$
where $C$ is a keyhole contour about the positive real axis. This contour integral is equal to (I omit the steps where I show the integral vanishes about the circular arcs)
$$-i 4 \pi \int_0^{\infty} dv \frac{8 (v^2-1)(v^4-6 v^2+1)}{v^8+4 v^6+70v^4+4 v^2+1} \log{v} + 4 \pi^2 \int_0^{\infty} dv \frac{8 (v^2-1)(v^4-6 v^2+1)}{v^8+4 v^6+70v^4+4 v^2+1}$$
It should be noted that the second integral vanishes; this may be easily seen by exploiting the symmetry about $v \mapsto 1/v$.
On the other hand, the contour integral is $i 2 \pi$ times the sum of the residues about the poles of the integrand. In general, this requires us to find the zeroes of the eight degree polynomial, which may not be possible analytically. Here, on the other hand, we have many symmetries to exploit, e.g., if $a$ is a root, then $1/a$ is a root, $-a$ is a root, and $\bar{a}$ is a root. For example, we may deduce that
$$z^8+4 z^6+70z^4+4 z^2+1 = (z^4+4 z^3+10 z^2+4 z+1) (z^4-4 z^3+10 z^2-4 z+1)$$
which exploits the $a \mapsto -a$ symmetry. Now write
$$z^4+4 z^3+10 z^2+4 z+1 = (z-a)(z-\bar{a})\left (z-\frac{1}{a}\right )\left (z-\frac{1}{\bar{a}}\right )$$
Write $a=r e^{i \theta}$ and get the following equations:
$$\left ( r+\frac{1}{r}\right ) \cos{\theta}=-2$$
$$\left (r^2+\frac{1}{r^2}\right) + 4 \cos^2{\theta}=10$$
From these equations, one may deduce that a solution is $r=\phi+\sqrt{\phi}$ and $\cos{\theta}=1/\phi$, where $\phi=(1+\sqrt{5})/2$ is the golden ratio. Thus the poles take the form
$$z_k = \pm \left (\phi\pm\sqrt{\phi}\right) e^{\pm i \arctan{\sqrt{\phi}}}$$
Now we have to find the residues of the integrand at these 8 poles. We can break this task up by computing:
$$\sum_{k=1}^8 \operatorname*{Res}_{z=z_k} \left [\frac{8 (z^2-1)(z^4-6 z^2+1) \log^2{z}}{z^8+4 z^6+70z^4+4 z^2+1}\right ]=\sum_{k=1}^8 \operatorname*{Res}_{z=z_k} \left [\frac{8 (z^2-1)(z^4-6 z^2+1)}{z^8+4 z^6+70z^4+4 z^2+1}\right ] \log^2{z_k}$$
Here things got very messy, but the result is rather unbelievably simple:
$$\operatorname*{Res}_{z=z_k} \left [\frac{8 (z^2-1)(z^4-6 z^2+1)}{z^8+4 z^6+70z^4+4 z^2+1}\right ] = \text{sgn}[\cos{(\arg{z_k})}]$$
EDIT
Actually, this is a very simple computation. Inspired by @sos440, one may express the rational function of $z$ in a very simple form:
$$\frac{8 (z^2-1)(z^4-6 z^2+1)}{z^8+4 z^6+70z^4+4 z^2+1} = -\left [\frac{p'(z)}{p(z)} + \frac{p'(-z)}{p(-z)} \right ]$$
where
$$p(z)=z^4+4 z^3+10 z^2+4 z+1$$
The residue of this function at the poles are then easily seen to be $\pm 1$ according to whether the pole is a zero of $p(z)$ or $p(-z)$.
END EDIT
That is, if the pole has a positive real part, the residue of the fraction is $+1$; if it has a negative real part, the residue is $-1$.
Now consider the log piece. Expanding the square, we get 3 terms:
$$\log^2{|z_k|} - (\arg{z_k})^2 + i 2 \log{|z_k|} \arg{z_k}$$
Summing over the residues, we find that because of the $\pm1$ contributions above, that the first and third terms sum to zero. This leaves the second term. For this, it is crucial that we get the arguments right, as $\arg{z_k} \in [0,2 \pi)$. Thus, we have
$$\begin{align}I= \int_0^{\infty} dv \frac{8 (v^2-1)(v^4-6 v^2+1)}{v^8+4 v^6+70v^4+4 v^2+1} \log{v} &= \frac12 \sum_{k=1}^8 \text{sgn}[\cos{(\arg{z_k})}] (\arg{z_k})^2 \\ &= \frac12 [2 (\arctan{\sqrt{\phi}})^2 + 2 (2 \pi - \arctan{\sqrt{\phi}})^2 \\ &- 2 (\pi - \arctan{\sqrt{\phi}})^2 - 2 (\pi + \arctan{\sqrt{\phi}})^2]\\ &= 2 \pi^2 -4 \pi \arctan{\sqrt{\phi}} \\ &= 4 \pi \, \text{arccot}{\sqrt{\phi}}\\\end{align}$$
Best Answer
A Neater Expression
$$I=\int^π_0 \frac {\sin x}{\sqrt {x^3+x+1}} dx = \sum_0^∞ A_k \sum_0^k (-1)^r {}^kP_{2r} π^{k-2r}$$ Where, $$(3+2k)A_k + (5+2k)A_{2+k} + 2(3+k)A_{3+k} = 0,$$ $$A_0=1, A_1=\frac {-1}{2}, A_2=\frac {3}{8}$$
A greedy approach
$$I=\int^π_0 \frac {\sin x}{\sqrt {x^3+x+1}} dx = 0.8750439062939084$$ Using the greedy Egyptian fraction algorithm, $$x_{k+1} = x_k - \frac {1}{\lceil \frac {1}{x_k} \rceil}$$ where, $x_0 = I$, I got an expansion, $$I = \frac {1}{2} + \frac {1}{3} + \frac {1}{2^3.3} +\frac {1}{2^3.3.13.73}+\frac {1}{2^2.13.113.397547} +……$$ I couldn't go farther , for my limited computational capacity (which is my laptop), however I indeed see one pattern : the prime factors in the denominators $(2,3,13,73,113,…)$ belong to the set of primes given by, $$a(n)= \text {Min} \left(x; π[x]-π\left[\frac {x}{2}\right]=n\right)$$ I got it on OEIS(https://oeis.org/A080359). Yet it needs much more insight.
Original answer
A closed form would be extremely difficult to get. This appears to be a new function. Substituting $t$ for the denominator, we get a beautiful form of the integral, however potentially latent in the present context. $$I = 2\int \frac {\cosh J(t)}{\cosh 3J(t)} \sin \left(-2 \sqrt {\frac {1}{3}} \sinh J(t)\right) dt$$ where, $$J(t):=\frac {1}{3} \sinh^{-1} \left[\frac {3\sqrt 3}{2} (1-t^2)\right]$$ So , I am giving a series form solution. Consider, $$F(x):=\int \frac {\sin x}{\sqrt {1+x(1+x^2)}} dx\tag{1}$$ Now, for $x<1$, $$[1+x(1+x^2)]^{-\frac {1}{2}} = -\sum_{k=0}^∞ C^{k-\frac {1}{2}}_{-\frac {1}{2}}x^k(1+x^2)^k \tag{2}$$ Plugging $(2)$ into $(1)$ we get, $$F(x) = -\sum_{k=0}^∞ C^{k-\frac {1}{2}}_{-\frac {1}{2}} G(k,x) \tag{3},$$ where , $$G(k,x):= \int x^k \sin x (1+x^2)^k dx \tag{4}$$ But, $$(1+x^2)^k= \sum_{r=0}^k C^k_r x^{2r} \tag{5}$$ Plugging $(5)$ into $(4)$ we get, $$G(k,x) = \sum_{r=0}^k C^k_r H(r,k,x) \tag{6},$$ where, $$H(r,k,x) := \int x^{k+2r} \sin x dx$$ $$= - \frac {\Gamma (k+2r+1, ix) + (-1)^{k+2r}\Gamma (k+2r+1, -ix)}{2(-1)^{\frac {5}{2} (k+2r)}} \tag{7}$$ Hence, $$F(x)=\sum_{k=0}^∞ C^{k-\frac {1}{2}}_{-\frac {1}{2}} \sum_{r=0}^k C^k_r \frac {\Gamma (k+2r+1, ix) + (-1)^{k+2r}\Gamma (k+2r+1, -ix)}{2(-1)^{\frac {5}{2} (k+2r)}}\tag{8}$$ On the same lines an integral exists for the case $x>1$, the only difference being in the binomial expansion for the denominator of the original integral . Call it $F'(x)$. Then, $$\int_0^π \frac {\sin x}{\sqrt {1+x(1+x^2)}} dx = [F(1) -F(0)] +[F'(π) - F'(1)]$$ This solution is in terms of upper incomplete gamma functions with complex arguments. The notation $C^n_r$ stands for combinatorial coefficients.
Note-1
Alternatively, one could use hypergeometric functions to express the final result, $$F(x)=-\sum_{k=0}^∞ C^{k-\frac {1}{2}}_{-\frac {1}{2}} \sum_{r=0}^k C^k_r \frac {x^{k+2r+2}}{k+2r+2} {}_1\text {F}_2 \left(\frac {k+2r+4}{2},\frac {3}{2} ; \frac {k+2r+2}{2} ; -\frac {x^2}{4}\right)$$
Note-2
There is still hope for a closed form of the indefinite integral in terms of the Fresnel integral $C(z)$, consider a related integral: $$\int \frac {\sin x}{\sqrt {x^3}} dx = \sqrt {8π} \text {C} \left(\sqrt {\frac {2x}{π}}\right) - \frac {2\sin x}{\sqrt x} + c$$ The integrand is almost the same, except for the extra linear term $x+1$ under square root.