For $\alpha, \beta, \gamma \in (0,1)$ satisfying $\alpha+\beta+\gamma = 1$ and
$\mu \in \mathbb{C} \setminus [1,\infty)$, define
$$
F_{\alpha\beta}(\mu) = \int_0^1\frac{dx}{x^\alpha(1-x)^\beta(1-\mu x)^\gamma}
\quad\text{ and }\quad
\Delta = \frac{\Gamma(1-\alpha)\Gamma(1-\beta)}{\Gamma(1+\gamma)}
$$
When $|\mu| < 1$, we can rewrite the integral $F_{\alpha\beta}(\mu)$ as
$$\begin{align}
F_{\alpha\beta}(\mu)
= & \int_0^1 \frac{1}{x^\alpha(1-x)^{\beta}}\left(\sum_{n=0}^{\infty}\frac{(\gamma)_n}{n!}\mu^n x^n\right) dx
= \sum_{n=0}^{\infty}\frac{(\gamma)_n}{n!}\frac{\Gamma(n+1-\alpha)\Gamma(1-\beta)}{\Gamma(n+1+\gamma)}\mu^n\\
= & \Delta\sum_{n=0}^{\infty}\frac{(\gamma)_n (1-\alpha)_n}{n!(\gamma+1)_n}\mu^n
= \Delta\gamma \sum_{n=0}^{\infty}\frac{(1-\alpha)_n}{n!(\gamma+n)}\mu^n
\end{align}$$
This implies
$$
\mu^{-\gamma} \left(\mu\frac{\partial}{\partial \mu}\right) \mu^{\gamma} F_{\alpha\beta}(\mu) =
\Delta\gamma \sum_{n=0}^{\infty}\frac{(1-\alpha)_n}{n!}\mu^n
= \Delta\gamma\frac{1}{(1-\mu)^{1-\alpha}}
$$
and hence
$$F_{\alpha\beta}(\mu)
= \Delta\gamma \mu^{-\gamma} \int_0^\mu \frac{\nu^{\gamma-1}d\nu}{(1-\nu)^{1-\alpha}}
= \Delta\gamma \int_0^1 \frac{t^{\gamma-1} dt}{(1-\mu t)^{1-\alpha}}
= \Delta \int_0^1 \frac{dt}{(1 - \mu t^{1/\gamma})^{1-\alpha}}$$
Notice if we substitute $x$ by $y = 1-x$, we have
$$F_{\alpha\beta}(\mu) = \int_0^1 \frac{dy}{y^\beta(1-y)^\alpha(1-\mu - \mu y)^{\gamma}}
= \frac{1}{(1-\mu)^\gamma} F_{\beta\alpha}(-\frac{\mu}{1-\mu})$$
Combine these two representations of $F_{\alpha\beta}(\mu)$ and let $\omega = \left(\frac{\mu}{1-\mu}\right)^{\gamma}$, we obtain
$$F_{\alpha\beta}(\mu) = \frac{\Delta}{(1-\mu)^{\gamma}}\int_0^1 \frac{dt}{( 1 + \omega^{1/\gamma} t^{1/\gamma})^{1-\beta}} = \frac{\Delta}{\mu^\gamma}\int_0^\omega \frac{dt}{(1 + t^{1/\gamma})^{1-\beta}}$$
Let $(\alpha,\beta,\gamma) = (\frac14,\frac12,\frac14)$ and $\mu = \frac{\sqrt{3}}{2}$, the identity we want to check becomes
$$\frac{\Gamma(\frac34)\Gamma(\frac12)}{\Gamma(\frac54) (\sqrt{3})^{1/4}}\int_0^\omega \frac{dt}{\sqrt{1+t^4}} \stackrel{?}{=} \frac{2\sqrt{2}}{3\sqrt[8]{3}} \pi\tag{*1}$$
Let $K(m)$ be the complete elliptic integral of the first kind associated with modulus $m$. i.e.
$$K(m) = \int_0^1 \frac{dx}{\sqrt{(1-x^2)(1-mx^2)}}$$
It is known that $\displaystyle K(\frac12) = \frac{8\pi^{3/2}}{\Gamma(-\frac14)^2}$. In term of $K(\frac12)$, it is easy to check $(*1)$ is equivalent to
$$\int_0^\omega \frac{dt}{\sqrt{1+t^4}} \stackrel{?}{=} \frac23 K(\frac12)\tag{*2}$$
To see whether this is the case, let $\varphi(u)$ be the inverse function of above integral.
More precisely, define $\varphi(u)$ by following relation:
$$u = \int_0^{\varphi(u)} \frac{dt}{\sqrt{1+t^4}}$$
Let $\psi(u)$ be $\frac{1}{\sqrt{2}}(\varphi(u) + \varphi(u)^{-1})$. It is easy to check/verify
$$
\varphi'(u)^2 = 1 + \varphi(u)^4
\implies
\psi'(u)^2 = 4 (1 - \psi(u)^2)(1 - \frac12 \psi(u)^2)
$$
Compare the ODE of $\psi(u)$ with that of a Jacobi elliptic functions with modulus $m = \frac12$, we find
$$\psi(u) = \text{sn}(2u + \text{constant} | \frac12 )\tag{*3}$$
Since we are going to deal with elliptic functions/integrals with $m = \frac12$ only,
we will simplify our notations and drop all reference to modulus, i.e
$\text{sn}(u)$ now means $\text{sn}(u|m=\frac12)$ and $K$ means $K(m = \frac12)$.
Over the complex plane, it is known that $\text{sn}(u)$ is doubly periodic with
fundamental period $4 K$ and $2i K$. It has two poles at $i K$ and $(2 + i)K$ in the fundamental domain.
When $u = 0$, we want $\varphi(u) = 0$ and hence $\psi(u) = \infty$. So the constant
in $(*3)$ has to be one of the pole. For small and positive $u$, we want $\varphi(u)$ and hence $\psi(u)$ to be positive. This fixes the constant to $i K$. i.e.
$$\psi(u) = \text{sn}(2u + iK )$$
and the condition $(*2)$ becomes whether following equality is true or not.
$$\frac{1}{\sqrt{2}} (\omega + \omega^{-1}) \stackrel{?}{=} \text{sn}( \frac43 K + i K)\tag{*4}$$
Notice $ 3( \frac43 K + i K) = 4 K + 3 i K $ is a pole of $\text{sn}(u)$. if one repeat
apply the addition formula for $\text{sn}(u+v)$
$$\text{sn}(u+v) = \frac{\text{sn}(u)\text{cn}(v)\text{dn}(v)+\text{sn}(v)\text{cn}(u)\text{dn}(u)}{1-m\,\text{sn}(u)^2 \text{sn}(v)^2}$$
One find in order for $\text{sn}(3u)$ to blow up, $\text{sn}(u)$ will be a root of
following polynomial equation:
$$3 m^2 s^8-4 m^2 s^6-4 m s^6+6 m s^4-1 = 0$$
Substitute $m = \frac12$ and $s = \frac{1}{\sqrt{2}}(t+\frac{1}{t})$ into this, the equation $\omega$ need to satisfy is given by:
$$(t^8 - 6 t^4 - 3)(3 t^8 + 6 t^4 - 1 ) = 0$$
One can check that $\omega = \sqrt[4]{\frac{\sqrt{3}}{2-\sqrt{3}}}$ is indeed a root of this polynomial. As a result, the original equality is valid.
Let's start with this Pfaff transformation for $\,a=\frac 14,b=\frac34,c=\frac23$ :
$$\tag{1}_2F_1\left(a,b\;;c\;;z\right)=(1-z)^{-a}\;_2F_1\left(a,c-b\;;c\;;\frac z{z-1}\right)$$
The 'Darboux evaluation' $(42)$ of Vidunas' "Transformations of algebraic Gauss hypergeometric functions" is :
$$\tag{2}_2F_1\left(\frac14,-\frac 1{12};\,\frac23;\,\frac {x(x+4)^3}{4(2x-1)^3}\right)=(1-2x)^{-1/4}$$
Solving $\,\displaystyle\frac {x(x+4)^3}{4(2x-1)^3}=\frac z{z-1}\,$ gives :
$$\tag{3}z=\frac{x(x+4)^3}{(x^2-10x-2)^2} $$
that we will use as :
$$\tag{4}z-1=\frac {4(2x-1)^3}{(x^2-10x-2)^2}$$
while $(1)$ and $(2)$ return :
$$_2F_1\left(\frac14,\frac34;\,\frac23;\,z\right)=\left[(z-1)(2x-1)\right]^{-1/4}$$
so that :
$$_2F_1\left(\frac14,\frac34;\,\frac23;\,z\right)=\left[\frac{4\,(2x-1)^4}{(x^2-10x-2)^2}\right]^{-1/4}$$
and (up to a minus sign) :
$$\tag{5}_2F_1\left(\frac14,\frac34;\,\frac23;\,z\right)^2=-\frac{x^2-10x-2}{2\,(2x-1)^2}$$
and indeed the substitution of $(z-1)$ and $_2F_1()^2$ with $(4)$ and $(5)$ in your formula gives :
$$27\,(z-1)^2\cdot{_2F_1}\left(\tfrac14,\tfrac34;\tfrac23;z\right)^8+18\,(z-1)\cdot{_2F_1}\left(\tfrac14,\tfrac34;\tfrac23;z\right)^4-8\cdot{_2F_1}\left(\tfrac14,\tfrac34;\tfrac23;z\right)^2=1$$
Many other formulae of this kind may be deduced using Vidunas' paper.
Best Answer
Let $t = 1 + y^3$, we can rewrite $B\left(\alpha^2; \frac12, \frac13 \right)$ as
$$\int_0^{\alpha^2} t^{-1/2} (1-t)^{-2/3} dt = \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{3y^2dy}{\sqrt{1+y^3} y^2} = 3 \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{dy}{\sqrt{1+y^3}}$$
Following the setup in my answer to a related question. Let $\;\displaystyle\eta = \frac{\Gamma\left(\frac13\right)\Gamma\left(\frac16\right)}{\sqrt{3\pi}}\;$ and $\wp(z)$ be the Weierstrass elliptic $\wp$ function with fundamental periods $1$ and $e^{i\pi/3}$. $\wp(z)$ is known to satisfy an ODE of the form $$\wp'(z)^2 = 4 \wp(z)^3 - g_2 \wp(z) - g_3\quad\text{ where }\quad g_2 = 0 \;\text{ and }\;g_3 = \frac{\eta^6}{16}$$ If one perform variable substitution $\;\displaystyle y = -\frac{4}{\eta^2} \wp\left(\frac{iz}{\eta}\right)$, one has
$$\frac{dy}{\sqrt{1+y^3}} = -dz\quad\text{ and }\quad \begin{cases} y\left(\frac{\sqrt{3}\eta}{3}\right) = -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{3}\right) = 0\\ \\ y\left(\frac{\sqrt{3}\eta}{2}\right) = -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{2}\right) = -1 \end{cases}$$ Using this, we can express conjecture $(8)$ in terms of $y(\cdot)$ and/or $\wp(\cdot)$: $$\begin{align} & B\left(\alpha^2; \frac12, \frac13 \right) \stackrel{?}{=} \frac{\sqrt{\pi}}{2}\frac{\Gamma\left(\frac13\right)}{\Gamma\left(\frac56\right)} = \frac{\sqrt{3}}{4}\eta\\ \iff & 3\left[y^{-1}(-1) - y^{-1}(-\sqrt[3]{1-\alpha^2})\right] \stackrel{?}{=} \frac{\sqrt{3}}{4}\eta\\ \iff & y^{-1}(-\sqrt[3]{1-\alpha^2}) \stackrel{?}{=} \frac{5\sqrt{3}}{12}\eta\\ \iff & \frac{4}{\eta^2}\wp\left(i\frac{5\sqrt{3}}{12}\right) \stackrel{?}{=} \sqrt[3]{1-\alpha^2} \end{align} $$ Let $u_0 = i\frac{\sqrt{3}}{3}$, $u_{-1} = i\frac{\sqrt{3}}{2}$ and $u = i\frac{5\sqrt{3}}{12} = \frac12(u_0 + u_{-1})$. Using the addition formula for $\wp$ function, we have
$$ \wp(2u) = \wp(u_0 + u_{-1}) = \frac14\left[\frac{\wp'(u_0)-\wp'(u_{-1})}{\wp(u_0)-\wp(u_{-1})}\right]^2 - \wp(u_0) - \wp(u_{-1})\\ =\frac14\left[\frac{-i\frac{\eta^3}{4} - 0}{0 - \frac{\eta^2}{4}}\right]^2 - 0 - \frac{\eta^2}{4} = -\frac{\eta^2}{2} $$ Using the duplication formula of $\wp$ function, we get $$ -\frac{\eta^2}{2} = \wp(2u) = \frac14\left(\frac{(6\wp(u)^2-\frac12 g_2)^2}{4\wp(u)^3-g_2\wp(u)-g_3}\right) -2\wp(u) = \frac{9\wp(u)^4}{4\wp(u)^3-\frac{\eta^4}{16}} - 2\wp(u) $$ Let $Y = \frac{4}{\eta^2}\wp(u)$ and $A^2 = 1 - Y^3$, above condition is equivalent to
$$\begin{align} & Y^4 + 8 Y^3 + 8 Y - 8 = 0\tag{*1a}\\ \iff & Y(8+Y^3) = 8(1-Y^3)\tag{*1b}\\ \implies & (9-A^2)^3(1-A^2) = 512A^6\tag{*1c}\\ \iff & (A^4-24A^3+18A^2-27)(A^4+24A^3+18A^2-27) = 0\tag{*1d} \end{align}$$
i.e. $Y = \sqrt[3]{1-\alpha^2}$ satisfies $(*1b)$ and hence $(*1a)$.