This question has been out for a while, and I think it deserves a thorough answer, even tho I came quite late to this party.
As both the classical Legendre-Jacobi theory and the Carlson theory have been mentioned by other users, I'll treat the OP's integral from both viewpoints.
Legendre-Jacobi
The OP came pretty close to using the correct substitution. One thing that could have been done instead is to recall the Pythagorean identity $1+\tan^2 u=\sec^2 u$, so that the substitution that should have been used is $x=a \tan^2 u$, where $a=2$ or $a=3$. Taking the smaller value of $a$, and after some amount of algebra, we obtain
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=\int_0^{\pi/2}\frac{2}{\sqrt{3-\sin^2u}}\mathrm du\\
&=\frac2{\sqrt{3}}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{1-\frac13\sin^2u}}\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where I use the parameter convention for elliptic integrals. (This is the same convention used in Abramowitz and Stegun. Relatedly, see here for an extended discussion on the notational confusion surrounding elliptic integrals.)
Had we chosen the substitution with $a=3$ instead, we would have instead obtained the result $\sqrt{2}K\left(-\frac12\right)$, which is equivalent through the imaginary modulus transformation
$$K(-m)=\frac1{\sqrt{1+m}}K\left(\frac{m}{m+1}\right),\quad m>0$$
Carlson
In Carlson's theory, there is the general hypergeometric function
$$R_{-a}(b_1,\dots,b_k;z_1,\dots,z_k)=\frac1{\mathbf B\left(a,-a+\sum_j b_j\right)}\int_0^\infty u^{-a-1+\sum_j b_j}\prod_j \left(u+z_j\right)^{-b_j}\mathrm du$$
where $\mathbf B(x,y)=\frac{\Gamma(x)\Gamma(y)}{\Gamma(x+y)}$ is the usual Euler beta function.
This multivariate hypergeometric function is a homogenized/symmetrized version of the classical Lauricella $F_D$ function (see e.g. this paper where Carlson introduced his function, tho that reference uses an opposite sign convention for $a$).
With this consideration, the OP's original integral can be expressed either as a three-variable Carlson integral, or a two-variable Carlson integral:
$$\begin{align*}\int_0^\infty\frac{\mathrm dx}{\sqrt{x(x+2)(x+3)}}&=2\,R_{-\frac12}\left(\frac12,\frac12,\frac12;0,2,3\right)\\&=\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)\end{align*}$$
The three-variable (incomplete) case occurs often enough that it is given the notation
$$\begin{align*}R_F(x,y,z)&=\frac12\int_0^\infty\frac{\mathrm du}{\sqrt{(u+x)(u+y)(u+z)}}\\&=R_{-\frac12}\left(\frac12,\frac12,\frac12;x,y,z\right)\end{align*}$$
and the two-variable form corresponds to the so-called "complete case",
$$\begin{align*}R_K(x,y)&=R_{-\frac12}\left(\frac12,\frac12;x,y\right)\\&=\frac2{\pi}R_F(0,x,y)\end{align*}$$
In fact, the two-variable form has the integral representation
$$R_K(x,y)=\frac2{\pi}\int_0^{\pi/2}\frac{\mathrm du}{\sqrt{x\cos^2 u+y\sin^2 u}}$$
which one recognizes to be related to the integral representation of Gauss's arithmetic-geometric mean (AGM):
$$R_K(x,y)=\frac1{\operatorname{agm}(\sqrt{x},\sqrt{y})}$$
Thus, the OP's integral is $\pi/\operatorname{agm}(\sqrt{2},\sqrt{3})$.
Additionally, the two-variable Carlson function is also related to the Gauss hypergeometric function, being a homogeneous version of it:
$$R_{-a}(b_1,b_2;x,y)=y^{-a}{}_2 F_1\left({{a,b_1}\atop{b_1+b_2}}\middle|1-\frac{x}{y}\right)$$
so one has
$$\begin{align*}\pi\,R_{-\frac12}\left(\frac12,\frac12;2,3\right)&=\frac{\pi}{\sqrt{3}}{}_2 F_1\left({{\frac12,\frac12}\atop{1}}\middle|\frac13\right)\\&=\frac2{\sqrt{3}}K\left(\frac13\right)\end{align*}$$
where we have used the hypergeometric representation of the complete elliptic integral of the first kind.
Best Answer
Google up "arithmetic geometric mean" or simply agm; there are also several related Q&A on mathoverflow.
Edit: Gerald Edgar remarks that I probably read your question the other way around. If this is case, then deducing the $j$ invariant (and hence $k$) from $K$ is even easier: as already alluded to in Felipe Voloch's comment, the $j$ invariant has a simple closed form in terms of theta constants, and theta constants have very fast converging expressions in terms of $q:=e^{2\pi i \tau}$.