In addition to the nice set of references by Raymond Manzoni, here is my proof of the identity. Frankly, I have not seen these references yet, thus I am not sure if this already appears in one of them.
Here I refer to the following identity
$$ \binom{\alpha}{\omega} = \frac{1}{2\pi} \int_{-\pi}^{\pi} \left(1 + e^{i\theta}\right)^{\alpha} e^{-i\omega \theta} \; d\theta, \ \cdots \ (1) $$
whose proof can be found in my blog post.
Now let $x$ be a real number such that $|x| < \frac{\pi}{2}$. Then simple calculation shows that
$$ \log\left(1+e^{2ix}\right) = \log(2\cos x) + ix \quad \Longleftrightarrow \quad \Im \left( \frac{-x}{\log\left(1+e^{2ix}\right)} \right) = \frac{x^2}{x^2 + \log^2(2\cos x)},$$
hence we have
$$ \begin{align*}I
&:= \int_{0}^{\frac{\pi}{2}} \frac{x^2}{x^2 + \log^2(2\cos x)} \; dx
= -\int_{0}^{\frac{\pi}{2}} \Im \left( \frac{x}{\log\left(1+e^{2ix}\right)} \right) \; dx \\
&= -\frac{1}{8}\int_{-\pi}^{\pi} \Im \left( \frac{\theta}{\log\left(1+e^{i\theta}\right)} \right) \; d\theta
= \frac{1}{8}\Re \left( \int_{-\pi}^{\pi} \frac{i\theta}{\log\left(1+e^{i\theta}\right)} \; d\theta \right).
\end{align*}$$
Differentiating both sides of $(1)$ with respect to $\omega$ and plugging $\omega = 1$, we have
$$ \frac{1}{2\pi} \int_{-\pi}^{\pi} (-i\theta) \left(1 + e^{i\theta}\right)^{\alpha} e^{-i\theta} \; d\theta = \alpha \left(\psi_0(\alpha) - \psi_0(2)\right). $$
Now integrating both sides with respect to $\alpha$ on $[0, 1]$,
$$ \begin{align*}
-\frac{1}{2\pi} \int_{-\pi}^{\pi} \frac{i\theta}{\log \left(1 + e^{i\theta}\right)} \; d\theta
&= \int_{0}^{1} \alpha \left(\psi_0(\alpha) - \psi_0(2)\right) \; d\alpha \\
&= \left[ \alpha \log \Gamma (\alpha) \right]_{0}^{1} - \int_{0}^{1} \log \Gamma (\alpha) \; d\alpha - \frac{1}{2}\psi_0(2) \\
&= -\frac{1}{2}\left( 1 - \gamma + \log (2\pi) \right),
\end{align*}$$
where we have used the fact that
$$ \psi_0 (1+n) = -\gamma + H_n, \quad n \in \mathbb{N}$$
and
$$ \begin{align*}
\int_{0}^{1} \log \Gamma (\alpha) \; d\alpha
& = \frac{1}{2} \int_{0}^{1} \log \left[ \Gamma (\alpha) \Gamma (1-\alpha) \right] \; d\alpha \\
&= \frac{1}{2} \int_{0}^{1} \log \left( \frac{\pi}{\sin \pi \alpha} \right) \; d\alpha \\
&= \frac{1}{2} \left( \log \pi - \int_{0}^{1} \log \sin \pi \alpha \; d\alpha \right) \\
&= \frac{1}{2} \log (2\pi).
\end{align*} $$
Therefore we have the desired result.
$$I:=\int_{0}^{\infty}\frac{\ln{(x)}}{\sqrt{x}\,\sqrt{x+1}\,\sqrt{2x+1}}\mathrm{d}x.$$
After first multiplying and dividing the integrand by 2, substitute $x=\frac{t}{2}$:
$$I=\int_{0}^{\infty}\frac{2\ln{(x)}}{\sqrt{2x}\,\sqrt{2x+2}\,\sqrt{2x+1}}\mathrm{d}x=\int_{0}^{\infty}\frac{\ln{\left(\frac{t}{2}\right)}}{\sqrt{t}\,\sqrt{t+2}\,\sqrt{t+1}}\mathrm{d}t.$$
Next, substituting $t=\frac{1}{u}$ yields:
$$\begin{align}
I
&=-\int_{0}^{\infty}\frac{\ln{(2u)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u\\
&=-\int_{0}^{\infty}\frac{\ln{(2)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u-\int_{0}^{\infty}\frac{\ln{(u)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u\\
&=-\int_{0}^{\infty}\frac{\ln{(2)}}{\sqrt{u}\sqrt{u+1}\sqrt{2u+1}}\mathrm{d}u-I\\
\implies I&=-\frac{\ln{(2)}}{2}\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}.
\end{align}$$
Making the sequence of substitutions $x=\frac{u-1}{2}$, then $u=\frac{1}{t}$, and finally $t=\sqrt{w}$, puts this integral into the form of a beta function:
$$\begin{align}
\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u-1}\sqrt{u+1}\sqrt{u}}\\
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u^2-1}\sqrt{u}}\\
&=\int_{1}^{0}\frac{t^{3/2}}{\sqrt{1-t^2}}\frac{(-1)}{t^2}\mathrm{d}t\\
&=\int_{0}^{1}\frac{\mathrm{d}t}{\sqrt{t}\,\sqrt{1-t^2}}\\
&=\frac12\int_{0}^{1}\frac{\mathrm{d}w}{w^{3/4}\,\sqrt{1-w}}\\
&=\frac12\operatorname{B}{\left(\frac14,\frac12\right)}\\
&=\frac12\frac{\Gamma{\left(\frac12\right)}\Gamma{\left(\frac14\right)}}{\Gamma{\left(\frac34\right)}}\\
&=\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}
\end{align}$$
Hence,
$$I=-\frac{\ln{(2)}}{2}\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}=-\frac{\pi^{3/2}\,\ln{(2)}}{2^{3/2}\,\Gamma^2{\left(\frac34\right)}}.~~~\blacksquare$$
Possible Alternative: You could also derive the answer from the complete elliptic integral of the first kind instead of from the beta function by making the substitution $t=z^2$ instead of $t=\sqrt{w}$.
$$\begin{align}
\int_{0}^{\infty}\frac{\mathrm{d}x}{\sqrt{x}\sqrt{x+1}\sqrt{2x+1}}
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u-1}\sqrt{u+1}\sqrt{u}}\\
&=\int_{1}^{\infty}\frac{\mathrm{d}u}{\sqrt{u^2-1}\sqrt{u}}\\
&=\int_{1}^{0}\frac{t^{3/2}}{\sqrt{1-t^2}}\frac{(-1)}{t^2}\mathrm{d}t\\
&=\int_{0}^{1}\frac{\mathrm{d}t}{\sqrt{t}\,\sqrt{1-t^2}}\\
&=2\int_{0}^{1}\frac{\mathrm{d}z}{\sqrt{1-z^4}}\\
&=2\,K{(-1)}\\
&=\frac{\Gamma^2{\left(\frac14\right)}}{2\sqrt{2\pi}}\\
&=\frac{\pi^{3/2}}{2^{1/2}\Gamma^2{\left(\frac34\right)}}.
\end{align}$$
Best Answer
Let $\alpha = a$, $$c_n = \frac{\Gamma\left(\frac{n+\alpha+1}{2}\right)}{\Gamma\left(\frac{n+\alpha+2}{2}\right)}$$ using the fact that $\Gamma(x+1) = x\Gamma(x)$, then $$c_{n+2} = \frac{n+\alpha+1}{n+\alpha+2}c_n$$
Let
$$f(x) = \sum_{n=0}^\infty (-1)^n c_n x^n$$
then,
$$f'(x) = \sum_{n=0}^\infty n(-1)^n c_n x^{n-1}$$
Start by proving that: $$\left(1-x^2\right) xf'(x) + \left(\alpha(1 - x^2) - x^2\right)f(x) = \beta - \gamma x$$ with \begin{align} \beta &= \alpha c_0\\ \gamma &= (\alpha + 1)c_1. \end{align} Indeed, \begin{align} \left(1-x^2\right) xf'(x) + \left(\alpha(1 - x^2) - x^2\right)f(x) &= \sum_{n=0}^{\infty} n(-1)^nc_nx^n - \sum_{n=0}^{\infty} n(-1)^nc_nx^{n+2} + \sum_{n=0}^{\infty} \alpha (-1)^n c_nx^n - \sum_{n=0}^{\infty} (\alpha+1) (-1)^n c_nx^{n+2}\\ &= \alpha c_0 -(\alpha+1)c_1 x + \sum_{n=0}^{\infty} (-1)^n \underbrace{\left((n+\alpha +2)c_{n+2} - (n+\alpha+1)c_n\right)}_{=0} x^{n+2}\\ &= \beta - \gamma x \end{align}
Let $g(x) = f(x)x^\alpha \sqrt{1 - x^2}$, then \begin{align} g'(x) &= f'(x) x^{\alpha}\sqrt{1-x^2} + \alpha f(x) x^{\alpha - 1}\sqrt{1-x^2} - f(x)x^{\alpha}\frac{x}{\sqrt{1-x^2}}\\ &= \frac{x^{\alpha - 1}}{\sqrt{1 - x^2}}\left(\left(1-x^2\right) xf'(x) + \left(\alpha(1 - x^2) - x^2\right)f(x)\right)\\ &=\frac{x^{\alpha - 1}}{\sqrt{1 - x^2}}\left(\beta - \gamma x\right) \end{align}
L'Hôpital's rule: since $g(1)=0$, \begin{align} f(1) &=\lim_{x\to 1} \frac{g(x)}{x^\alpha \sqrt{1-x^2}}\\ &=\lim_{x\to 1} \frac{g'(x)}{\displaystyle\alpha x^{\alpha-1}\sqrt{1-x^2} - x^{\alpha} \frac{x}{\sqrt{1-x^2}}}\\ &= \lim_{x\to 1} \frac{\displaystyle\left(\beta -\gamma x\right) \frac{x^{\alpha -1}}{\sqrt{1-x^2}}}{\displaystyle\alpha x^{\alpha-1}\sqrt{1-x^2} - x^{\alpha} \frac{x}{\sqrt{1-x^2}}}\\ &= \lim_{x\to 1} \frac{\displaystyle\left(\beta - \gamma x\right)}{\displaystyle\alpha \left(1-x^2\right) - x^2}\\ &= \gamma - \beta. \end{align}
To finish the proof you need to simplify $\gamma - \beta$ which is not difficult.