I am trying to find for $a>b>0$ :
$$I=\int_{0}^{\pi }{\frac{x\sin \left( x \right)}{a+b{{\cos }^{2}}\left( x \right)}dx}$$
I don't think a substitution is an option here, besides differentiation under the sign integral makes the integral harder. Fortunately Mathematica gives a closed form for the integral in question:
$$\frac{\pi \tan^{-1}\left( \sqrt{\frac{b}{a}} \right)}{\sqrt{ab}}$$
Best Answer
HINT:
Enforce the substitution $x\mapsto \pi-x$.
Then, note that
$$I=\int_0^\pi \frac{x\sin(x)}{a+b\cos^2(x)}\,dx=\int_0^\pi \frac{(\pi-x)\sin(x)}{a+b\cos^2(x)}\,dx$$
Can you finish now?