$\newcommand{\bbx}[1]{\,\bbox[15px,border:1px groove navy]{\displaystyle{#1}}\,}
\newcommand{\braces}[1]{\left\lbrace\,{#1}\,\right\rbrace}
\newcommand{\bracks}[1]{\left\lbrack\,{#1}\,\right\rbrack}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\ds}[1]{\displaystyle{#1}}
\newcommand{\expo}[1]{\,\mathrm{e}^{#1}\,}
\newcommand{\ic}{\mathrm{i}}
\newcommand{\mc}[1]{\mathcal{#1}}
\newcommand{\mrm}[1]{\mathrm{#1}}
\newcommand{\on}[1]{\operatorname{#1}}
\newcommand{\pars}[1]{\left(\,{#1}\,\right)}
\newcommand{\partiald}[3][]{\frac{\partial^{#1} #2}{\partial #3^{#1}}}
\newcommand{\root}[2][]{\,\sqrt[#1]{\,{#2}\,}\,}
\newcommand{\totald}[3][]{\frac{\mathrm{d}^{#1} #2}{\mathrm{d} #3^{#1}}}
\newcommand{\verts}[1]{\left\vert\,{#1}\,\right\vert}$
First step: Multiply numerator and denominator by
$\ds{\sec^{2m + 2n}\pars{\theta} =
\sec^{2m - 1}\pars{\theta}\
\sec^{2n - 1}\pars{\theta}\
\sec^{2}\pars{\theta}}$.
\begin{align}
&\bbox[5px,#ffd]{\int_{0}^{\pi/2}
{\sin^{2m - 1}\pars{\theta}\cos^{2n - 1}\pars{\theta} \over \bracks{a\sin^{2}\pars{\theta} + b\cos^{2}\pars{\theta}}^{m + n}}
\,\dd\theta}
\\[5mm] = &
\int_{0}^{\pi/2}
{\tan^{2m - 1}\pars{\theta} \over
\bracks{a\tan^{2}\pars{\theta} + b}^{m + n}}
\sec^{2}\pars{\theta}\,\dd\theta
\\[5mm] \stackrel{x\ =\ \tan\pars{\theta}}{=}\,\,\,&
\int_{0}^{\infty}{x^{2m - 1} \over
\pars{ax^{2} + b}^{m + n}}\,\dd x
\\[5mm] \stackrel{x^{2}\ \mapsto\ x}{=}\,\,\,&
{1 \over 2}\int_{0}^{\infty}{x^{m - 1} \over
\pars{ax + b}^{m + n}}\,\dd x
\\[5mm] \stackrel{ax/b\ \mapsto\ x}{=}\,\,\,&
{1 \over 2a^{m}b^{n}}\
\underbrace{\int_{0}^{\infty}{x^{m - 1} \over \pars{x + 1}^{m + n}}\,\dd x}_{\ds{\on{B}\pars{m,n}}}
\end{align}
See this link.
Then,
\begin{align}
&\bbox[5px,#ffd]{\int_{0}^{\pi/2}
{\sin^{2m - 1}\pars{\theta}\cos^{2n - 1}\pars{\theta} \over \bracks{a\sin^{2}\pars{\theta} + b\cos^{2}\pars{\theta}}^{m + n}}
\,\dd\theta}
\\[5mm] = &\
\bbx{{1 \over 2a^{m}b^{n}}\,{\Gamma\pars{m}\Gamma\pars{n}
\over \Gamma\pars{m + n}}} \\ &
\end{align}
Your method was correct, you just made a small mistake in the simplification. It is true that
\begin{align}
\int_0^{\dfrac{\pi}{2}} \sqrt{\sin\theta}\mathrm d\theta \int_0^{\dfrac{\pi}{2}} \frac{1}{\sqrt{\sin \theta}} \mathrm d \theta
\end{align}
equals
\begin{align}
& \beta(\frac{3}{4}, \frac{1}{2})\; \beta(\frac{1}{4}, \frac{1}{2}) \\
\\
&=\frac{\Gamma(\frac{3}{4})\Gamma(\frac{1}{2})}{2\Gamma(\frac{3}{4}+\color{red}{\frac{1}{2}})}\cdot \frac{\Gamma(\frac{1}{4})\Gamma(\frac{1}{2})}{2\Gamma(\frac{1}{4}+\frac{1}{2})} \\
\\
&=\frac{\Gamma(\frac{3}{4})\Gamma(\frac{1}{2})}{2\Gamma(\frac{5}{4})} \cdot \frac{\Gamma(\frac{1}{4}) \Gamma(\frac{1}{2})}{2\Gamma(3/4)}
\end{align}
which is the step from your book. Then using $\Gamma(\frac{1}{2})\Gamma(\frac{1}{2}) = \Gamma(\frac{1}{2})\Gamma(1-\frac{1}{2}) = \frac{\pi}{sin(\pi / 2)} = \pi$ is correct, and will lead to the answer.
Best Answer
We know that $$\int_0^{2a} f(x)dx=2\int_0^af(x)dx$$
if $f(a+x)=f(a-x)$
here $f(\theta)=\sin^5(\theta)\cos^4\theta$
Check $f(\pi/2+\theta)=f(\pi/2-\theta)$.
So, $$\int_0^{{2\cdot\frac{\pi}{2}}}\sin^5(\theta)\cos^4(\theta)d\theta=2\int_0^{\pi/2}\sin^5(\theta)\cos^4(\theta)d\theta$$
Rest is just application of Walli's formula.