Thank you for posting this question, I enjoyed trying to answer it.
Start with the expression that Mathematica gave you and replace each
argument $\frac14$ of a hypergeometric function with $\frac z4$,
because we will be taking limits. I will call the two hypergeometric
functions $Q_1(z)$ and $Q_2(z)$. Each term can be brought to a closed
form by using identity 16.6.2 from the DLMF.
Setting $a=\frac12$, $b=1-\frac\nu2$, we get
$$ Q_1(z) = (1-z)^{-\frac12} F\left(\frac16,
\frac36,\frac 56; 1-\frac\nu2, 1+\frac\nu2; \frac{-27 z}{4(1-z)^3}
\right), $$
and setting $a=\frac{1+3\nu}{2}$, $b=1+\frac\nu2$, we get
$$ Q_2(z) = (1-z)^{-\frac{1+3\nu}{2}} F\left(\frac{1+3\nu}6,
\frac{3+3\nu}{6}, \frac{5+3\nu}{6}; 1+\frac\nu2, 1+\nu;
\frac{-27z}{4(1-z)^3} \right). $$
(Note that there are 6 possible identities to try per function, one for each possible choice of
$a$ and $b$ from the parameters, so it helps to do this on a
computer.)
The reason this works is that now the point $z=1$ is a singular point
of the hypergeometric functions on the right hand side, and Mathematica
will succeed in finding the limits as $z\to1$. The expression for the
whole integral that you have is
$$ Q =
\frac{2^{\frac43}\pi^{\frac12}}{\Gamma(-\frac16)\Gamma(\frac56+\frac\nu2)\Gamma(\frac56-\frac\nu2)\sin\frac{\nu\pi}2}
\left( -1 + 3^{-\frac{3\nu}2}\cos\left(\frac{\nu\pi}{2}\right)
\frac{\Gamma(\frac{1+3\nu}{2})
\Gamma(\frac56-\frac\nu2)}{\Gamma(\frac{1+\nu}2)\Gamma(\frac56+\frac\nu2)}
\right). $$
Call the large expression in brackets $A$, and then write
$$ A = -1 + B 3^{-\frac{3\nu}{2}}\cos\frac{\pi\nu}{2}, \qquad B = \frac{\Gamma(\frac{1+3\nu}{2})
\Gamma(\frac56-\frac\nu2)}{\Gamma(\frac{1+\nu}2)\Gamma(\frac56+\frac\nu2)}. $$
Now, Mathematica will not simplify $A$ or $B$ on its own, so it needs
help. Set $x=\frac16+\frac\nu2$, and use the multiplication formula to
get
$$
\frac{\Gamma(\frac{1+3\nu}2)}{\Gamma(\frac{1+\nu}{2})\Gamma(\frac56+\frac\nu2)}
= \frac{\Gamma(3x)}{\Gamma(x+\frac13)\Gamma(x+\frac23)} =
\frac{\Gamma(x)}{2\pi} 3^{3x-1/2}. $$
After this, $A$ simplifies to
$$ A = -1 +
\frac{\Gamma(\frac16+\frac\nu2)\Gamma(\frac56-\frac\nu2)}{2\pi}\cos\frac{\pi\nu}{2}
= -1 + \frac{\cos\frac{\pi\nu}{2}}{2\sin(\frac\pi6+\frac{\pi\nu}{2})},
$$
where I've also used the reflection formula for $\Gamma(z)\Gamma(1-z)$ to get rid of the
gamma functions. Some further amount of manual trigonometry yields
$$ A = -\frac{\sqrt{3}}{2}\frac{\sin\frac{\pi\nu}{2}}{\sin(\frac\pi6 +
\frac{\nu\pi}{2})}. $$
Finally, write
$$ \frac{1}{\sin(\frac\pi6+\frac{\pi\nu}{2})} =
\frac{\Gamma(\frac16+\frac\nu2)\Gamma(\frac56-\frac\nu2)}{\pi}, $$
and substitute back. Lots of things cancel, and the answer is
$$ Q = -\frac{3^{1/2}2^{1/3}}{\pi^{1/2}}
\frac{\Gamma(\frac16+\frac\nu2)}{\Gamma(-\frac16)\Gamma(\frac56+\frac\nu2)}. $$
This closed form is equivalent to the one you gave through the
use of $\Gamma(\frac16)\Gamma(-\frac16)=-12\pi$.
P.S.
I would also like to note that the integral
$$ I(\nu,c) = \int_0^\infty J_\nu(x)^2 J_\nu(c x)\,dx $$
and its general form
$$ \int_0^\infty x^{\rho-1}J_\nu(a x) J_\mu(b x) J_\lambda(c x)\,dx $$
appear in Gradshteyn and Ryzhik, and you can find a paper "Some
infinite integrals involving bessel functions, I and II" by
W. N. Bailey, which evaluates this integral in terms of Appell
functions, but only in the case $c>2$ ($|c|>|a|+|b|$), which is where the $F_4$ Appell
function converges. DLMF 16.16.6 actually gives a way to write this
integral as
$$
\frac{\Gamma(\frac{1+3\nu}{2})c^{-1-2\nu}}{\Gamma(1+\nu)^2\Gamma(\frac{1-\nu}{2})}
\,\,\,{}_2F_1\left( \frac{1+\nu}{2}, \frac{1+3\nu}{2}; 1+\nu; x
\right)^2, \qquad x = \frac{1-\sqrt{1-4/c^2}}{2}, $$
but the issue is that this is only correct for $c>2$, and the rhs is
complex for $c<2$. Appell function would only be defined by analytic
continuation in this case anyway, and I didn't find anything useful about
non-principal branches of Appell or hypergeometric functions.
For $c>2$, Mathematica also gives the following:
$$
I(\nu,c) = \frac{\Gamma(\frac{1+3\nu}{2})c^{-1-2\nu}}{\Gamma(\frac{1-\nu}{2})\Gamma(1+\nu)^2}
\,\,\,{}_3F_2\left( \frac{1+\nu}{2}, \frac{1}{2}+\nu,
\frac{1+3\nu}{2}; 1+\nu, 1+2\nu; \frac{4}{c^2} \right), $$
but this is incorrect when $c<2$.
Let $t = 1 + y^3$, we can rewrite $B\left(\alpha^2; \frac12, \frac13 \right)$ as
$$\int_0^{\alpha^2} t^{-1/2} (1-t)^{-2/3} dt
= \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{3y^2dy}{\sqrt{1+y^3} y^2}
= 3 \int_{-1}^{-\sqrt[3]{1-\alpha^2}} \frac{dy}{\sqrt{1+y^3}}$$
Following the setup in my answer to a related question. Let
$\;\displaystyle\eta = \frac{\Gamma\left(\frac13\right)\Gamma\left(\frac16\right)}{\sqrt{3\pi}}\;$ and $\wp(z)$ be the Weierstrass elliptic $\wp$ function with fundamental periods $1$ and $e^{i\pi/3}$. $\wp(z)$ is known to satisfy an ODE of the form
$$\wp'(z)^2 = 4 \wp(z)^3 - g_2 \wp(z) - g_3\quad\text{ where }\quad g_2 = 0 \;\text{ and }\;g_3 = \frac{\eta^6}{16}$$
If one perform variable substitution $\;\displaystyle y = -\frac{4}{\eta^2} \wp\left(\frac{iz}{\eta}\right)$, one has
$$\frac{dy}{\sqrt{1+y^3}} = -dz\quad\text{ and }\quad
\begin{cases}
y\left(\frac{\sqrt{3}\eta}{3}\right)
= -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{3}\right) = 0\\
\\
y\left(\frac{\sqrt{3}\eta}{2}\right)
= -\frac{4}{\eta^2}\wp\left(i\frac{\sqrt{3}}{2}\right) = -1
\end{cases}$$
Using this, we can express conjecture $(8)$ in terms of $y(\cdot)$ and/or $\wp(\cdot)$:
$$\begin{align}
& B\left(\alpha^2; \frac12, \frac13 \right)
\stackrel{?}{=} \frac{\sqrt{\pi}}{2}\frac{\Gamma\left(\frac13\right)}{\Gamma\left(\frac56\right)} = \frac{\sqrt{3}}{4}\eta\\
\iff & 3\left[y^{-1}(-1) - y^{-1}(-\sqrt[3]{1-\alpha^2})\right]
\stackrel{?}{=} \frac{\sqrt{3}}{4}\eta\\
\iff & y^{-1}(-\sqrt[3]{1-\alpha^2})
\stackrel{?}{=} \frac{5\sqrt{3}}{12}\eta\\
\iff & \frac{4}{\eta^2}\wp\left(i\frac{5\sqrt{3}}{12}\right)
\stackrel{?}{=} \sqrt[3]{1-\alpha^2}
\end{align}
$$
Let $u_0 = i\frac{\sqrt{3}}{3}$, $u_{-1} = i\frac{\sqrt{3}}{2}$ and $u = i\frac{5\sqrt{3}}{12} = \frac12(u_0 + u_{-1})$. Using the addition formula for $\wp$ function,
we have
$$
\wp(2u) = \wp(u_0 + u_{-1})
= \frac14\left[\frac{\wp'(u_0)-\wp'(u_{-1})}{\wp(u_0)-\wp(u_{-1})}\right]^2 - \wp(u_0) - \wp(u_{-1})\\
=\frac14\left[\frac{-i\frac{\eta^3}{4} - 0}{0 - \frac{\eta^2}{4}}\right]^2 - 0 - \frac{\eta^2}{4}
= -\frac{\eta^2}{2}
$$
Using the duplication formula of $\wp$ function, we get
$$
-\frac{\eta^2}{2} = \wp(2u) = \frac14\left(\frac{(6\wp(u)^2-\frac12 g_2)^2}{4\wp(u)^3-g_2\wp(u)-g_3}\right) -2\wp(u)
= \frac{9\wp(u)^4}{4\wp(u)^3-\frac{\eta^4}{16}} - 2\wp(u)
$$
Let $Y = \frac{4}{\eta^2}\wp(u)$ and $A^2 = 1 - Y^3$, above condition is equivalent to
$$\begin{align}
& Y^4 + 8 Y^3 + 8 Y - 8 = 0\tag{*1a}\\
\iff & Y(8+Y^3) = 8(1-Y^3)\tag{*1b}\\
\implies & (9-A^2)^3(1-A^2) = 512A^6\tag{*1c}\\
\iff & (A^4-24A^3+18A^2-27)(A^4+24A^3+18A^2-27) = 0\tag{*1d}
\end{align}$$
- Since $\alpha$ is a root for one of the factors in $(*1d)$, $A = \alpha$ satisfies $(*1d)$ and hence $(*1c)$.
- Since $0 < \alpha < 1$ implies $1 - \alpha^2 > 0$, $(*1c) \implies (*1b)$ in this particular case.
i.e. $Y = \sqrt[3]{1-\alpha^2}$ satisfies $(*1b)$ and hence $(*1a)$.
- Since $u$ lies between $u_0$ and $u_{-1}$, $\frac{4}{\eta^2}\wp(u) > 0$. Using
the fact $(*1a)$ has only one positive root, we find $\frac{4}{\eta^2}\wp(u) = \sqrt[3]{1-\alpha^2}$. i.e. conjecture $(8)$ is true.
Best Answer
For $\alpha, \beta, \gamma \in (0,1)$ satisfying $\alpha+\beta+\gamma = 1$ and $\mu \in \mathbb{C} \setminus [1,\infty)$, define
$$ F_{\alpha\beta}(\mu) = \int_0^1\frac{dx}{x^\alpha(1-x)^\beta(1-\mu x)^\gamma} \quad\text{ and }\quad \Delta = \frac{\Gamma(1-\alpha)\Gamma(1-\beta)}{\Gamma(1+\gamma)} $$ When $|\mu| < 1$, we can rewrite the integral $F_{\alpha\beta}(\mu)$ as
$$\begin{align} F_{\alpha\beta}(\mu) = & \int_0^1 \frac{1}{x^\alpha(1-x)^{\beta}}\left(\sum_{n=0}^{\infty}\frac{(\gamma)_n}{n!}\mu^n x^n\right) dx = \sum_{n=0}^{\infty}\frac{(\gamma)_n}{n!}\frac{\Gamma(n+1-\alpha)\Gamma(1-\beta)}{\Gamma(n+1+\gamma)}\mu^n\\ = & \Delta\sum_{n=0}^{\infty}\frac{(\gamma)_n (1-\alpha)_n}{n!(\gamma+1)_n}\mu^n = \Delta\gamma \sum_{n=0}^{\infty}\frac{(1-\alpha)_n}{n!(\gamma+n)}\mu^n \end{align}$$ This implies $$ \mu^{-\gamma} \left(\mu\frac{\partial}{\partial \mu}\right) \mu^{\gamma} F_{\alpha\beta}(\mu) = \Delta\gamma \sum_{n=0}^{\infty}\frac{(1-\alpha)_n}{n!}\mu^n = \Delta\gamma\frac{1}{(1-\mu)^{1-\alpha}} $$ and hence $$F_{\alpha\beta}(\mu) = \Delta\gamma \mu^{-\gamma} \int_0^\mu \frac{\nu^{\gamma-1}d\nu}{(1-\nu)^{1-\alpha}} = \Delta\gamma \int_0^1 \frac{t^{\gamma-1} dt}{(1-\mu t)^{1-\alpha}} = \Delta \int_0^1 \frac{dt}{(1 - \mu t^{1/\gamma})^{1-\alpha}}$$
Notice if we substitute $x$ by $y = 1-x$, we have
$$F_{\alpha\beta}(\mu) = \int_0^1 \frac{dy}{y^\beta(1-y)^\alpha(1-\mu - \mu y)^{\gamma}} = \frac{1}{(1-\mu)^\gamma} F_{\beta\alpha}(-\frac{\mu}{1-\mu})$$
Combine these two representations of $F_{\alpha\beta}(\mu)$ and let $\omega = \left(\frac{\mu}{1-\mu}\right)^{\gamma}$, we obtain
$$F_{\alpha\beta}(\mu) = \frac{\Delta}{(1-\mu)^{\gamma}}\int_0^1 \frac{dt}{( 1 + \omega^{1/\gamma} t^{1/\gamma})^{1-\beta}} = \frac{\Delta}{\mu^\gamma}\int_0^\omega \frac{dt}{(1 + t^{1/\gamma})^{1-\beta}}$$
Let $(\alpha,\beta,\gamma) = (\frac14,\frac12,\frac14)$ and $\mu = \frac{\sqrt{3}}{2}$, the identity we want to check becomes
$$\frac{\Gamma(\frac34)\Gamma(\frac12)}{\Gamma(\frac54) (\sqrt{3})^{1/4}}\int_0^\omega \frac{dt}{\sqrt{1+t^4}} \stackrel{?}{=} \frac{2\sqrt{2}}{3\sqrt[8]{3}} \pi\tag{*1}$$
Let $K(m)$ be the complete elliptic integral of the first kind associated with modulus $m$. i.e.
$$K(m) = \int_0^1 \frac{dx}{\sqrt{(1-x^2)(1-mx^2)}}$$ It is known that $\displaystyle K(\frac12) = \frac{8\pi^{3/2}}{\Gamma(-\frac14)^2}$. In term of $K(\frac12)$, it is easy to check $(*1)$ is equivalent to
$$\int_0^\omega \frac{dt}{\sqrt{1+t^4}} \stackrel{?}{=} \frac23 K(\frac12)\tag{*2}$$
To see whether this is the case, let $\varphi(u)$ be the inverse function of above integral. More precisely, define $\varphi(u)$ by following relation:
$$u = \int_0^{\varphi(u)} \frac{dt}{\sqrt{1+t^4}}$$
Let $\psi(u)$ be $\frac{1}{\sqrt{2}}(\varphi(u) + \varphi(u)^{-1})$. It is easy to check/verify $$ \varphi'(u)^2 = 1 + \varphi(u)^4 \implies \psi'(u)^2 = 4 (1 - \psi(u)^2)(1 - \frac12 \psi(u)^2) $$
Compare the ODE of $\psi(u)$ with that of a Jacobi elliptic functions with modulus $m = \frac12$, we find
$$\psi(u) = \text{sn}(2u + \text{constant} | \frac12 )\tag{*3}$$
Since we are going to deal with elliptic functions/integrals with $m = \frac12$ only, we will simplify our notations and drop all reference to modulus, i.e $\text{sn}(u)$ now means $\text{sn}(u|m=\frac12)$ and $K$ means $K(m = \frac12)$.
Over the complex plane, it is known that $\text{sn}(u)$ is doubly periodic with fundamental period $4 K$ and $2i K$. It has two poles at $i K$ and $(2 + i)K$ in the fundamental domain. When $u = 0$, we want $\varphi(u) = 0$ and hence $\psi(u) = \infty$. So the constant in $(*3)$ has to be one of the pole. For small and positive $u$, we want $\varphi(u)$ and hence $\psi(u)$ to be positive. This fixes the constant to $i K$. i.e.
$$\psi(u) = \text{sn}(2u + iK )$$
and the condition $(*2)$ becomes whether following equality is true or not.
$$\frac{1}{\sqrt{2}} (\omega + \omega^{-1}) \stackrel{?}{=} \text{sn}( \frac43 K + i K)\tag{*4}$$
Notice $ 3( \frac43 K + i K) = 4 K + 3 i K $ is a pole of $\text{sn}(u)$. if one repeat apply the addition formula for $\text{sn}(u+v)$
$$\text{sn}(u+v) = \frac{\text{sn}(u)\text{cn}(v)\text{dn}(v)+\text{sn}(v)\text{cn}(u)\text{dn}(u)}{1-m\,\text{sn}(u)^2 \text{sn}(v)^2}$$
One find in order for $\text{sn}(3u)$ to blow up, $\text{sn}(u)$ will be a root of following polynomial equation: $$3 m^2 s^8-4 m^2 s^6-4 m s^6+6 m s^4-1 = 0$$ Substitute $m = \frac12$ and $s = \frac{1}{\sqrt{2}}(t+\frac{1}{t})$ into this, the equation $\omega$ need to satisfy is given by:
$$(t^8 - 6 t^4 - 3)(3 t^8 + 6 t^4 - 1 ) = 0$$
One can check that $\omega = \sqrt[4]{\frac{\sqrt{3}}{2-\sqrt{3}}}$ is indeed a root of this polynomial. As a result, the original equality is valid.