The general method of attack on an integral like this is to recognize that the $x^2$ could be ignored for the time being upon expressing the integral in terms of a suitable parameter which may then be twice differentiated. To wit, after a little manipulatin, I find:
$$\displaystyle \int_{0}^{\frac{\pi}{2}} dx \: {x}^2 \sqrt{\tan x} \sin \left( 2x \right) = 2 \Im{ \int_{0}^{\frac{\pi}{2}} dx \: x^2 e^{i x} \sqrt{\sin{x} \cos{x}}} $$
So consider the following integral:
$$J(\alpha) = \int_{0}^{\frac{\pi}{2}} dx \: e^{i \alpha x} \sqrt{\sin{x} \cos{x}} $$
Note that the integral we seek is
$$\displaystyle \int_{0}^{\frac{\pi}{2}} dx \: {x}^2 \sqrt{\tan x} \sin \left( 2x \right) = -2 \Im{\left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1}} $$
It turns out that there is, in fact, a closed for for $J(\alpha)$:
$$J(\alpha) = -\frac{\left(\frac{1}{16}+\frac{i}{16}\right) \sqrt{\frac{\pi }{2}} \left ( e^{\frac{i \pi a}{2}}-i\right) \Gamma \left(\frac{a-1}{4}\right)}{\Gamma \left ( \frac{a+5}{4} \right )} $$
Plugging this into the above expression, you will find terms including polylogs and harmonic numbers, which I will spare you unless explicitly asked for. But this is how you would get a closed-form expression for your integral.
EDIT
The integral is even nicer when we consider
$$J(\alpha) = \int_{0}^{\frac{\pi}{2}} dx \: \sin{\alpha x} \sqrt{\sin{x} \cos{x}} $$
Then
$$J(\alpha) = \frac{\pi ^{3/2} \sin \left(\frac{\pi a}{4}\right)}{8 \Gamma
\left(\frac{5-a}{4}\right) \Gamma \left(\frac{5+a}{4}\right)}$$
and
$$\left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1} = \frac{\pi \left(-5 \pi ^2+6 \pi (\log (4)-2)+3 (8+(\log (4)-4) \log
(4))\right)}{192 \sqrt{2}}$$
The integral we seek is then
$$\int_0^{\frac{\pi}{2}} dx \: x^2 \sqrt{\tan{x}} \sin{(2 x)} = -2 \left [ \frac{\partial^2}{\partial \alpha^2} J(\alpha) \right ]_{\alpha = 1} = \frac{\pi \left(-24+12 \pi +5 \pi ^2-3 \log ^2(4)+12 \log (4)-6 \pi \log
(4)\right)}{96 \sqrt{2}}$$
It turns out that the numerical value of the latter value is about $1.10577$, which agrees with the numerical approximation of the integral mentioned by @mrf and verified in Mathematica.
The integral equals $\sqrt{2+\sqrt{8}} \cdot \Omega$, where $\Omega$
is the real half-period $\Omega = 1.3736768699491\ldots$
of the elliptic curve
$$
E : y^2 = x^3 - 4 x^2 - 4 x,
$$
i.e. the complete elliptic integral
$$
\Omega = \int_{2-\sqrt{8}}^0 \frac{dx}{\sqrt{x^3-4x^2-4x}}
= \int_{2+\sqrt{8}}^\infty \frac{dx}{\sqrt{x^3-4x^2-4x}}
$$
(the integrand can also be brought to the classical form
${\bf K}(k) = \int_0^1 dz \, / \sqrt{(1-z^2) (1-k^2 z^2)}$,
but with a more complicated $k$ and probably also
an elementary factor more complicated than our $\sqrt{2+\sqrt{8}}$).
Here's gp code for this formula:
sqrt(2+sqrt(8)) * ellinit([0,-4,0,-4,0])[15]
The curve $E$ is reasonably nice, with conductor $128=2^7$ and
$j$-invariant $10976 = 2^5 7^3 = 1728 + 2^5 17^2$;
but $E$ does not have complex multiplication (CM), so we
do not expect to get a simpler form as would be possible for a CM curve
[e.g. $\int_1^\infty dx/\sqrt{x^3-1}$ is a Beta integral, and
$\int_0^\infty dx/\sqrt{x^3+4x^2+2x}
= \Gamma(1/8) \Gamma(3/8) / (4\sqrt{\pi})$].
Harry Peter already used the trigonometric substitution
$$
(\cos \phi, \sin \phi, d\phi) =
\left( \frac{1-t^2}{1+t^2}, \frac{2t}{1+t^2}, \frac{2 \, dt}{1+t^2} \right)
$$
(which I guess is the "Weierstrass substitution" suggested in the comment of
Steven Stadnicki) to write $I$ as
$$
\int_0^1 \frac{(1+t)dt}{\sqrt{(1+2t-t^2) (t-t^3)}},
$$
which is a half-period of the holomorphic differential
$(1+t) dt/u$ on the hyperelliptic curve $C: u^2 = (1+2t-t^2) (t-t^3)$
of genus $2$. Most such periods cannot be simplified further,
but this one is special because the curve has more symmetry than
just the "hyperelliptic involution" $(t,u) \leftrightarrow (t,-u)$.
In particular $C$ has an involution
$$
\iota: (t,u) \leftrightarrow
\left( \frac{1-t}{1+t}, \frac{2^{3/2}}{(1+t)^3} u \right)
$$
which also sends the interval $(0,1)$ to itself, reversing
the orientation. This suggests splitting the integral
at the midpoint $t_0 := \sqrt{2} - 1$ and applying
the change of variable $(t,dt) \leftarrow ((1-t)/(1+t), -2\,dt/(1+t)^2)$
to the integral over $(t_0,1)$ to obtain $\sqrt{2} \int_0^{t_0} dt/u$.
Hence
$$
I = \int_0^{t_0} \frac{(\sqrt{2}+1+t)dt}{\sqrt{(1+2t-t^2) (t-t^3)}}
$$
and now the change of variable $X = t + (1-t)/(1+t)$ transforms $I$ to an
elliptic integral corresponding to the quotient curve $C\,/\langle\iota\rangle$.
While $C\,/\langle\iota\rangle$ has irrational coefficients involving $\sqrt{2}$,
it has rational $j$-invariant, so we can find coordinates that identify
$C\,/\langle\iota\rangle$ with our curve $E$ with rational coefficients,
though at the cost of introducing the factor $\sqrt{2+\sqrt{8}}$
into the formula for $I$ given at the start of this answer.
Best Answer
\begin{align} \int_0^{\Large\frac{\pi}{4}} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx&=\int_0^1\frac{\sqrt{y(1-y)}}{1+y^2}\,dy\quad\Rightarrow\quad y=\tan x\\ &=\int_0^\infty\frac{\sqrt{t}}{(1+t)(1+2t+2t^2)}\,dt\quad\Rightarrow\quad t=\frac{y}{1-y}\\ &=\int_0^\infty\frac{2z^2}{(1+z^2)(1+2z^2+2z^4)}\,dz\quad\Rightarrow\quad z^2=t\\ &=2\int_0^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\ &=\int_{-\infty}^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\ &=I_1+I_2-\pi \end{align}
\begin{align} I_1 &=\int_{-\infty}^\infty\frac{2z^2}{1+2z^2+2z^4}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{z^2+\frac{1}{2z^2}+1}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{\left(z-\frac{1}{\sqrt{2}z}\right)^2+1+\sqrt{2}}\,dz\\ &=\int_{-\infty}^\infty\frac{1}{z^2+1+\sqrt{2}}\,dz\\ &=\frac{\pi}{\sqrt{1+\sqrt{2}}} \end{align} where the 4th line we use identity
The proof can be seen in my answer here. $I_2$ can be proved in similar manner (see user111187's answer). \begin{equation} I_2=\frac{1}{2}\int_{-\infty}^\infty\frac{1}{z^4+z^2+\frac{1}{2}}\,dz=\pi\sqrt{\frac{\sqrt{2}-1}{2}} \end{equation}
Combine all the results together, we finally get