In this case, the following trick also works: Dividing both the numerator and the denominator by $\cos^4 x$, we can use the substitution $ t = \tan x$ to obtain
\begin{align*}
\int_{0}^{\frac{\pi}{2}} \frac{dx}{(a^2 \cos^2 x + b^2 \sin^2 x)^2}
&= \int_{0}^{\frac{\pi}{2}} \frac{1 + \tan^2 x}{(a^2 + b^2 \tan^2 x)^2} \sec^2 x \, dx \\
&= \int_{0}^{\infty} \frac{1 + t^2}{(a^2 + b^2 t^2)^2} \, dt \\
&= \frac{1}{a^2}\int_{0}^{\infty} \left( \frac{1}{a^2 + b^2 t^2} + \frac{(a^2 - b^2) t^2}{(a^2 + b^2 t^2)^2} \right) \, dt.
\end{align*}
The first one can be evaluated as follows: Let $bt = a \tan\varphi$. Then
$$ \int_{0}^{\infty} \frac{dt}{a^2 + b^2 t^2} = \frac{1}{ab} \int_{0}^{\frac{\pi}{2}} d\varphi = \frac{\pi}{2ab}. $$
For the second one, we perform the integration by parts:
\begin{align*}
\int_{0}^{\infty} \frac{t^2}{(a^2 + b^2 t^2)^2} \, dt
&= \left[ - \frac{1}{b^2}\frac{1}{a^2 + b^2 t^2} \cdot \frac{t}{2} \right]_{0}^{\infty} + \int_{0}^{\infty} \frac{1}{2b^2}\frac{dt}{a^2 + b^2 t^2} \\
&= \frac{1}{2b^2} \int_{0}^{\infty} \frac{dt}{a^2 + b^2 t^2} \\
&= \frac{\pi}{4ab^3}.
\end{align*}
Putting together, the answer is
$$ \frac{(a^2 + b^2)\pi}{4(ab)^3}. $$
If you don't mind, I would like to present an alternative approach that makes use of the fact that
$$\int^\infty_0\frac{x^{p-1}}{1+x}dx=\frac{\pi}{\sin{p\pi}}$$
Simply factorise the denominator and decompose the integrand into partial fractions.
\begin{align}
\int^\infty_0\frac{x^a}{x^2+2(\cos{b})x+1}dx
&=\int^\infty_0\frac{x^a}{(x+e^{ib})(x+e^{-ib})}dx\\
&=\frac{1}{-e^{ib}+e^{-ib}}\int^\infty_0\frac{x^a}{e^{ib}+x}dx+\frac{1}{-e^{-ib}+e^{ib}}\int^\infty_0\frac{x^a}{e^{-ib}+x}dx\\
&=\frac{1}{-2i\sin{b}}\int^\infty_0\frac{(e^{ib}u)^a}{1+u}du+\frac{1}{2i\sin{b}}\int^\infty_0\frac{(e^{-ib}u)^a}{1+u}du\\
&=\frac{e^{iab}}{-2i\sin{b}}\frac{\pi}{\sin(\pi a+\pi)}+\frac{e^{-iab}}{2i\sin{b}}\frac{\pi}{\sin(\pi a+\pi)}\\
&=\frac{\pi}{\sin \pi a\sin{b}}\left(\frac{e^{iab}-e^{-iab}}{2i}\right)\\
&=\frac{\pi\sin{ab}}{\sin{\pi a}\sin{b}}
\end{align}
Best Answer
Using this, we have $$ \log(\sin(x))=-\log 2-\sum_{k=1}^{+\infty}\frac{\cos(2kx)}{k} $$ Therefore, $$ \int_0^{\pi/2}x^2\log(\sin x)dx=-\frac{\pi^3}{24}\log 2-\sum_{k=1}^{+\infty}\frac{1}{k}\int_0^{\pi/2}x^2\cos(2kx)dx $$ Using integration by parts, we have $$ \int_0^{\pi/2}x^2\cos(2kx)dx = \frac{(-1)^k\pi}{4k^2} $$ Thus, $$ \sum_{k=1}^{+\infty}\frac{1}{k}\int_0^{\pi/2}x^2\cos(2kx)dx=\frac{\pi}{4}\sum_{k=1}^{+\infty}\frac{(-1)^k}{k^3}=-\frac{\pi}{4}\eta(3)=-\frac{3\pi}{16}\zeta(3) $$ where $\eta$ is the Dirichlet eta function. Finally, $$ \int_0^{\pi/2}x^2\log(\sin x)dx=\frac{3\pi}{16}\zeta(3)-\frac{\pi^3}{24}\log 2 $$