This question has been out for a while, and I think it deserves a thorough answer, even tho I came quite late to this party.
As both the classical Legendre-Jacobi theory and the Carlson theory have been mentioned by other users, I'll treat the OP's integral from both viewpoints.
Legendre-Jacobi
The OP came pretty close to using the correct substitution. One thing that could have been done instead is to recall the Pythagorean identity $1+\tan^2 u=\sec^2 u$, so that the substitution that should have been used is $x=a \tan^2 u$, where $a=2$ or $a=3$. Taking the smaller value of $a$, and after some amount of algebra, we obtain
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=\int_0^{\pi/2}\frac{2}{\sqrt{3-\sin^2u}}\mathrm du\\
&=\frac2{\sqrt{3}}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{1-\frac13\sin^2u}}\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where I use the parameter convention for elliptic integrals. (This is the same convention used in Abramowitz and Stegun. Relatedly, see here for an extended discussion on the notational confusion surrounding elliptic integrals.)
Had we chosen the substitution with $a=3$ instead, we would have instead obtained the result $\sqrt{2}K\left(-\frac12\right)$, which is equivalent through the imaginary modulus transformation
$$K(-m)=\frac1{\sqrt{1+m}}K\left(\frac{m}{m+1}\right),\quad m>0$$
Carlson
In Carlson's theory, there is the general hypergeometric function
$$R_{-a}(b_1,\dots,b_k;z_1,\dots,z_k)=\frac1{\mathbf B\left(a,-a+\sum_j b_j\right)}\int_0^\infty u^{-a-1+\sum_j b_j}\prod_j \left(u+z_j\right)^{-b_j}\mathrm du$$
where $\mathbf B(x,y)=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$ is the usual Euler beta function.
This multivariate hypergeometric function is a homogenized/symmetrized version of the classical Lauricella $F_D$ function (see e.g. this paper where Carlson introduced his function, tho that reference uses an opposite sign convention for $a$).
With this consideration, the OP's original integral can be expressed either as a three-variable Carlson integral, or a two-variable Carlson integral:
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=2\,R_{-\frac12}\left(\frac12,\frac12,\frac12;0,2,3\right)\\&=\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)\end{align*}$$
The three-variable (incomplete) case occurs often enough that it is given the notation
$$\begin{align*}R_F(x,y,z)&=\frac12\int_0^\infty\frac{\mathrm du}{\sqrt{(u+x)(u+y)(u+z)}}\\&=R_{-\frac12}\left(\frac12,\frac12,\frac12;x,y,z\right)\end{align*}$$
and the two-variable form corresponds to the so-called "complete case",
$$\begin{align*}R_K(x,y)&=R_{-\frac12}\left(\frac12,\frac12;x,y\right)\\&=\frac2{\pi}R_F(0,x,y)\end{align*}$$
In fact, the two-variable form has the integral representation
$$R_K(x,y)=\frac2{\pi}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{x\cos^2 u+y\sin^2 u}}$$
which one recognizes to be related to the integral representation of Gauss's arithmetic-geometric mean (AGM):
$$R_K(x,y)=\frac1{\operatorname{agm}(\sqrt{x},\sqrt{y})}$$
Thus, the OP's integral is $\pi/\operatorname{agm}(\sqrt{2},\sqrt{3})$.
Additionally, the two-variable Carlson function is also related to the Gauss hypergeometric function, being a homogeneous version of it:
$$R_{-a}(b_1,b_2;x,y)=y^{-a}{}_2 F_1\left({{a,b_1}\atop{b_1+b_2}}\middle|1-\frac{x}{y}\right)$$
so one has
$$\begin{align*}\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)&=\frac{\pi}{\sqrt{3}}{}_2 F_1\left({{\frac12,\frac12}\atop{1}}\middle|\frac13\right)\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where we have used the hypergeometric representation of the complete elliptic integral of the first kind.
Here is an outline of the approach I have taken to solve this integral.
First rewrite the integral $(1)$ in Cartesian variables:
$$I=\int_{-\infty}^{\infty} \mathrm{d}^3v~ \frac{3x^2y^2v_z^2}{y^2v_x^2+x^2v_y^2+x^2y^2v_z^2}\cos(uv_x)\exp\left(-\frac{v_x^2}{2}-\frac{v_y^2}{2}-\frac{v_z^2}{2}\right). $$
Now use the following substitution
$$ \frac{1}{y^2v_x^2+x^2v_y^2+x^2y^2v_z^2}=\int_{0}^{\infty}d\tau~\mathrm{ e^{-(y^2v_x^2+x^2v_y^2+x^2y^2v_z^2)\tau}},$$
such that
$$ I=\int_{0}^{\infty}d\tau\int_{-\infty}^{\infty} \mathrm{d}^3v~3x^2y^2v_z^2\cos(uv_x) \mathrm{e^{-v_x^2(\tau y^2+1/2)-v_y^2(\tau x^2+1/2)-v_z^2(\tau x^2y^2+1/2)}}. $$
The $(v_x,v_y,v_z)$ can be evaluated with the help of Mathematica. The results gives
$$ \int_{-\infty}^{\infty} \mathrm{d}^3v~3x^2y^2v_z^2\cos(uv_x) \mathrm{e^{-\alpha v_x^2-\beta v_y^2-\gamma v_z^2}}=\frac{3\pi^{3/2}}{2x^2y^2}\frac{\exp\left(-\frac{u^2}{4y^2}\frac{1}{\tau+1/2y^2}\right)}{\left(\tau+1/2x^2y^2\right)^{3/2}\left(\tau+1/2x^2\right)^{1/2}\left(\tau+1/2y^2\right)^{1/2}}.$$
Thereby,
$$ I= -\frac{3~\mathrm{const}}{x^2y^2~\mathrm{const}}\int_{0}^{\infty}\mathrm{d\tau}\frac{\exp\left(-\frac{u^2}{4y^2}\frac{1}{\tau+1/2y^2}\right)}{\left(\tau+1/2x^2y^2\right)^{3/2}\left(\tau+1/2x^2\right)^{1/2}\left(\tau+1/2y^2\right)^{1/2}}. $$
Now, performing the substitution $\tau=\frac{1-x^2}{2x^2y^2k^2}-\frac{1}{2x^2y^2}$ gives us
$$ I=(\mathrm{const})~\frac{3x~y}{(1-x^2)^{3/2}}\int_{0}^{\sqrt{1-x^2}}\mathrm{dk}\frac{k^2\exp\left(-\frac{u^2}{2}\frac{x^2k^2}{\left(1-x^2\right)\left(1-k^2\right)}\right)}{\sqrt{1-k^2}\sqrt{1-k^2\frac{1-y^2}{1-x^2}}}, $$
which is the desired integral unless of a constant. ;)
Best Answer
Taking it up from Matt F.'s comment, the substitution $$\theta=g(\alpha):= \left(\alpha -\frac{\pi }{4}\right)1\left(\alpha \ge\frac{\pi }{4}\right)+\left(\alpha +\frac{\pi }{4}\right) 1\left(\alpha < \frac{\pi }{4}\right)$$ in the first integral transforms it to the second one.
The transformation $g\colon[0,\pi/2)\to[0,\pi/2)$ is a bijection preserving the Lebesgue measure.
Here is the image of Mathematica notebook confirming/illustrating what was said above: