Let $t = 1 + y^3$, we can rewrite $B\left(\alpha^2; \frac12, \frac13 \right)$ as
$$\int_0^{\alpha^2} t^{-1/2} (1-t)^{-2/3} dt
= \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{3y^2dy}{\sqrt{1+y^3} y^2}
= 3 \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{dy}{\sqrt{1+y^3}}$$
Following the setup in my answer to a related question. Let
$\;\displaystyle\eta = \frac{\Gamma\left(\frac13\right)\Gamma\left(\frac16\right)}{\sqrt{3\pi}}\;$ and $\wp(z)$ be the Weierstrass elliptic $\wp$ function with fundamental periods $1$ and $e^{i\pi/3}$. $\wp(z)$ is known to satisfy an ODE of the form
$$\wp'(z)^2 = 4 \wp(z)^3 - g_2 \wp(z) - g_3\quad\text{ where }\quad g_2 = 0 \;\text{ and }\;g_3 = \frac{\eta^6}{16}$$
If one perform variable substitution $\;\displaystyle y = -\frac{4}{\eta^2} \wp\left(\frac{iz}{\eta}\right)$, one has
$$\frac{dy}{\sqrt{1+y^3}} = -dz\quad\text{ and }\quad
\begin{cases}
y\left(\frac{\sqrt{3}\eta}{3}\right)
= -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{3}\right) = 0\\
\\
y\left(\frac{\sqrt{3}\eta}{2}\right)
= -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{2}\right) = -1
\end{cases}$$
Using this, we can express conjecture $(8)$ in terms of $y(\cdot)$ and/or $\wp(\cdot)$:
$$\begin{align}
& B\left(\alpha^2; \frac12, \frac13 \right)
\stackrel{?}{=} \frac{\sqrt{\pi}}{2}\frac{\Gamma\left(\frac13\right)}{\Gamma\left(\frac56\right)} = \frac{\sqrt{3}}{4}\eta\\
\iff & 3\left[y^{-1}(-1) - y^{-1}(-\sqrt[3]{1-\alpha^2})\right]
\stackrel{?}{=} \frac{\sqrt{3}}{4}\eta\\
\iff & y^{-1}(-\sqrt[3]{1-\alpha^2})
\stackrel{?}{=} \frac{5\sqrt{3}}{12}\eta\\
\iff & \frac{4}{\eta^2}\wp\left(i\frac{5\sqrt{3}}{12}\right)
\stackrel{?}{=} \sqrt[3]{1-\alpha^2}
\end{align}
$$
Let $u_0 = i\frac{\sqrt{3}}{3}$, $u_{-1} = i\frac{\sqrt{3}}{2}$ and $u = i\frac{5\sqrt{3}}{12} = \frac12(u_0 + u_{-1})$. Using the addition formula for $\wp$ function,
we have
$$
\wp(2u) = \wp(u_0 + u_{-1})
= \frac14\left[\frac{\wp'(u_0)-\wp'(u_{-1})}{\wp(u_0)-\wp(u_{-1})}\right]^2 - \wp(u_0) - \wp(u_{-1})\\
=\frac14\left[\frac{-i\frac{\eta^3}{4} - 0}{0 - \frac{\eta^2}{4}}\right]^2 - 0 - \frac{\eta^2}{4}
= -\frac{\eta^2}{2}
$$
Using the duplication formula of $\wp$ function, we get
$$
-\frac{\eta^2}{2} = \wp(2u) = \frac14\left(\frac{(6\wp(u)^2-\frac12 g_2)^2}{4\wp(u)^3-g_2\wp(u)-g_3}\right) -2\wp(u)
= \frac{9\wp(u)^4}{4\wp(u)^3-\frac{\eta^4}{16}} - 2\wp(u)
$$
Let $Y = \frac{4}{\eta^2}\wp(u)$ and $A^2 = 1 - Y^3$, above condition is equivalent to
$$\begin{align}
& Y^4 + 8 Y^3 + 8 Y - 8 = 0\tag{*1a}\\
\iff & Y(8+Y^3) = 8(1-Y^3)\tag{*1b}\\
\implies & (9-A^2)^3(1-A^2) = 512A^6\tag{*1c}\\
\iff & (A^4-24A^3+18A^2-27)(A^4+24A^3+18A^2-27) = 0\tag{*1d}
\end{align}$$
- Since $\alpha$ is a root for one of the factors in $(*1d)$, $A = \alpha$ satisfies $(*1d)$ and hence $(*1c)$.
- Since $0 < \alpha < 1$ implies $1 - \alpha^2 > 0$, $(*1c) \implies (*1b)$ in this particular case.
i.e. $Y = \sqrt[3]{1-\alpha^2}$ satisfies $(*1b)$ and hence $(*1a)$.
- Since $u$ lies between $u_0$ and $u_{-1}$, $\frac{4}{\eta^2}\wp(u) > 0$. Using
the fact $(*1a)$ has only one positive root, we find $\frac{4}{\eta^2}\wp(u) = \sqrt[3]{1-\alpha^2}$. i.e. conjecture $(8)$ is true.
$$I:=\int_{0}^{\infty}\frac{\ln{(x)}}{\sqrt{x}\,\sqrt{x+1}\,\sqrt{2x+1}}\mathrm{d}x.$$
After first multiplying and dividing the integrand by 2, substitute $x=\frac{t}{2}$:
$$I=\int_{0}^{\infty}\frac{2\ln{(x)}}{\sqrt{2x}\,\sqrt{2x+2}\,\sqrt{2x+1}}\mathrm{d}x=\int_{0}^{\infty}\frac{\ln{\left(\frac{t}{2}\right)}}{\sqrt{t}\,\sqrt{t+2}\,\sqrt{t+1}}\mathrm{d}t.$$
Next, substituting $t=\frac{1}{u}$ yields:
$$\begin{align}
I
&=-\int_{0}^{\infty}\frac{\ln{(2u)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u\\
&=-\int_{0}^{\infty}\frac{\ln{(2)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u-\int_{0}^{\infty}\frac{\ln{(u)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u\\
&=-\int_{0}^{\infty}\frac{\ln{(2)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u-I\\
\implies I&=-\frac{\ln{(2)}}{2}\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}.
\end{align}$$
Making the sequence of substitutions $x=\frac{u-1}{2}$, then $u=\frac{1}{t}$, and finally $t=\sqrt{w}$, puts this integral into the form of a beta function:
$$\begin{align}
\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u-1}\sqrt{u+1}\sqrt{u}}\\
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u^2-1}\sqrt{u}}\\
&=\int_{1}^{0}\frac{t^{3/2}}{\sqrt{1-t^2}}\frac{(-1)}{t^2}\mathrm{d}t\\
&=\int_{0}^{1}\frac{\mathrm{d}t}{\sqrt{t}\,\sqrt{1-t^2}}\\
&=\frac12\int_{0}^{1}\frac{\mathrm{d}w}{w^{3/4}\,\sqrt{1-w}}\\
&=\frac12\operatorname{B}{\left(\frac14,\frac12\right)}\\
&=\frac12\frac{\Gamma{\left(\frac12\right)}\Gamma{\left(\frac14\right)}}{\Gamma{\left(\frac34\right)}}\\
&=\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}
\end{align}$$
Hence,
$$I=-\frac{\ln{(2)}}{2}\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}=-\frac{\pi^{3/2}\,\ln{(2)}}{2^{3/2}\,\Gamma^2{\left(\frac34\right)}}.~~~\blacksquare$$
Possible Alternative: You could also derive the answer from the complete elliptic integral of the first kind instead of from the beta function by making the substitution $t=z^2$ instead of $t=\sqrt{w}$.
$$\begin{align}
\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u-1}\sqrt{u+1}\sqrt{u}}\\
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u^2-1}\sqrt{u}}\\
&=\int_{1}^{0}\frac{t^{3/2}}{\sqrt{1-t^2}}\frac{(-1)}{t^2}\mathrm{d}t\\
&=\int_{0}^{1}\frac{\mathrm{d}t}{\sqrt{t}\,\sqrt{1-t^2}}\\
&=2\int_{0}^{1}\frac{\mathrm{d}z}{\sqrt{1-z^4}}\\
&=2\,K{(-1)}\\
&=\frac{\Gamma^2{\left(\frac14\right)}}{2\sqrt{2\pi}}\\
&=\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}.
\end{align}$$
Best Answer
Consider the hypergeometric equation with parameters $(a,b,c)=\left(\frac16,\frac12,\frac13\right)$, and build from its two canonical solutions near $z=0$ the vector $$\vec{y}(z)=\left(\begin{array}{c} y_1 \\ y_2 \end{array}\right)=\left(\begin{array}{c} _2F_1(a,b;c;z) \\ z^{1-c}{}_2F_1(a-c+1,b-c+1;2-c;z) \end{array}\right).\tag{1}$$ This is a single-valued vector function on $\mathbb{C}\backslash\{(-\infty,0]\cup[1,\infty)\}$. Its analytic continuation along a closed loop $\gamma$ gives rise to monodromy representation of $\pi_1(\mathbb{C}\backslash\{0,1\})$: $$ \gamma\mapsto M_{[\gamma]},\qquad y(\gamma z)=M_{[\gamma]}y(z).$$ The monodromy group $G\subset GL(2,\mathbb{C})$ of the hypergeometric equation is generated by two matrices corresponding to simple loops around $0$ and $1$. In the case we are interested in these matrices are explicitly given by $$M_0=\left(\begin{array}{cc} 1 & 0 \\ 0 & e^{-2\pi i /3}\end{array}\right),\qquad M_1=C\left(\begin{array}{cc} 1 & 0 \\ 0 & e^{2\pi i /3}\end{array}\right)C^{-1},\tag{2}$$ where the connection matrix $C=\left(\begin{array}{cc} 1 & 2^{\frac43} \\ -2^{\frac83} & 8\end{array}\right)$. If $G$ is finite, then $\vec{y}(z)$ has a finite number of branches, and moreover (Schwarz 1872), is algebraic.
It is not difficult to check that the monodromy group $G$ generated by $M_0$, $M_1$ from (2) is indeed finite. In particular, note that $$M_0^3=M_1^3=I,\qquad M_1^2=-M_0M_1M_0, $$ $$M_1M_0M_1=-M_0^2,\qquad M_1M_0^2M_1=M_0^2M_1M_0^2.$$ It turns out that $G$ has order $24$ and is isomorphic to the binary tetrahedral group: $$G\cong 2T=\langle s,t\,|\,(st)^2=s^3=t^3\rangle, $$ where the generators can be identified as $s=M_0M_1M_0M_1M_0$, $t=M_1M_0M_1M_0M_1$.
Corollary: The hypergeometric functions in (1) are algebraic.
Algebraic solutions of the hypergeometric equations are classified by the so-called Schwarz table, and have been studied by many mathematicians, see e.g. the bibliography in this paper by R.Vidunas. Their explicit construction is somewhat involved but relatively straightforward - at least when the corresponding algebraic curve has genus $0$ (the genus can be determined independently from the Riemann-Hurwitz formula).
In our case the task simplifies even more as our parameter values can be obtained from the genus $0$ tetrahedral formula (2.4) of the above mentioned paper by a combination of a linear trasformation (sending $\frac56$ to $\frac43-\frac56=\frac12$) and differentiation (transforming $\frac43$ into $\frac13$). The result is $$_2F_1\left(\frac16,\frac12;\frac13;-\frac{r(r+2)^3}{(r+1)(1-r)^3}\right)=\frac{\sqrt{1-r^2}}{2r+1}.$$
Corollary: The antiderivative $\displaystyle\int \mathcal{R}\left(x,y(x)\right)dx$, where $y(x)={}_2F_1\left(\frac16,\frac12;\frac13;-x\right)$ and $\mathcal{R}(x,y)$ is rational in both arguments, can be expressed in terms of elementary functions.
Example: The transformation $r\mapsto x(r)=\frac{r(r+2)^3}{(r+1)(1-r)^3}$ bijectively maps $(0,1)$ to $(0,\infty)$, and therefore the initial integral becomes \begin{align}\mathcal{I}&=\int_0^1 \left(\frac{\sqrt{1-r^2}}{2r+1}\right)^{12}\left(\frac{r(r+2)^3}{(r+1)(1-r)^3}\right)'dr=\\&= 2\int_0^1 \frac{(1+r)^4(1-r)^2(r+2)^2}{(2r+1)^{10}} dr=\\&=\frac{80\,663}{153\,090}. \end{align}