\begin{align}
\int_0^{\Large\frac{\pi}{4}} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx&=\int_0^1\frac{\sqrt{y(1-y)}}{1+y^2}\,dy\quad\Rightarrow\quad y=\tan x\\
&=\int_0^\infty\frac{\sqrt{t}}{(1+t)(1+2t+2t^2)}\,dt\quad\Rightarrow\quad t=\frac{y}{1-y}\\
&=\int_0^\infty\frac{2z^2}{(1+z^2)(1+2z^2+2z^4)}\,dz\quad\Rightarrow\quad z^2=t\\
&=2\int_0^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\
&=\int_{-\infty}^\infty\left[\frac{2z^2}{1+2z^2+2z^4}+\frac{1}{1+2z^2+2z^4}-\frac{1}{1+z^2}\right]\,dz\\
&=I_1+I_2-\pi
\end{align}
\begin{align}
I_1
&=\int_{-\infty}^\infty\frac{2z^2}{1+2z^2+2z^4}\,dz\\
&=\int_{-\infty}^\infty\frac{1}{z^2+\frac{1}{2z^2}+1}\,dz\\
&=\int_{-\infty}^\infty\frac{1}{\left(z-\frac{1}{\sqrt{2}z}\right)^2+1+\sqrt{2}}\,dz\\
&=\int_{-\infty}^\infty\frac{1}{z^2+1+\sqrt{2}}\,dz\\
&=\frac{\pi}{\sqrt{1+\sqrt{2}}}
\end{align}
where the 4th line we use identity
\begin{align}
\int_{-\infty}^\infty f\left(x\right)\,dx=\int_{-\infty}^\infty f\left(x-\frac{a}{x}\right)\,dx\qquad,\qquad\text{for }\, a>0.
\end{align}
The proof can be seen in my answer here. $I_2$ can be proved in similar manner (see user111187's answer).
\begin{equation}
I_2=\frac{1}{2}\int_{-\infty}^\infty\frac{1}{z^4+z^2+\frac{1}{2}}\,dz=\pi\sqrt{\frac{\sqrt{2}-1}{2}}
\end{equation}
Combine all the results together, we finally get
\begin{equation}
\int_0^{\Large\frac{\pi}{4}} \sqrt{\tan x} \sqrt{1-\tan x}\,\,dx=\frac{\pi}{\sqrt[4]{2}}\sqrt{\frac{2+\sqrt{2}}{2}}-\pi
\end{equation}
Best Answer
hint
Your series is $$\sum_{n=0}^{+\infty}(x^{5n}-x^{5n+3})=$$
$$(1-x^3)\sum_{n=0}^{+\infty}(x^5)^n$$