I would like to compute $I$, using a different method, other than below
$$I=\large\int_{0}^{\pi/2}\csc^2(x)\ln\left(\frac{a+b\sin^2 x}{a-b\sin^2 x}\right)\mathrm dx$$
$a\ge b$
Applying integration by parts:
$\large u=\ln\left(\frac{a+b\sin^2 x}{a-b\sin^2 x}\right)$
$\large dv=\csc^2(x)dx$
$$I=2a\int_{0}^{\pi/2}\frac{\cot^2 x}{a\cot^2 x-b+a}\cdot \frac{\csc^2 x}{a\cot^2 x +b+a}\mathrm dx$$
Making a sub of $y=\cot x$ and using partial fractions decomposition and so on to solve for I.
Finally arrived at $$I=\pi\cdot \frac{\sqrt{a+b}-\sqrt{a-b}}{\sqrt{a}}$$
Best Answer
As suggested in comments by @Dahaka
Let $I(b)$ represent the given integral. Hence we have $I(0)=0$.
Now differentiating $I(b)$ with respect to $b$ we get $$I'(b) =\int_0^{\infty} \csc ^2x \cdot \frac {a-b\sin^2x}{a+b\sin^2x}\cdot\left( \frac {\sin^2x(a-b\sin^2x)+\sin^2x(a+b\sin^2x)}{(a-b\sin^2x)^2}\right) dx$$
Simplifying we get $$I'(b) =\int_0^{\infty} \left(\frac {1}{a-b\sin^2x} -\frac {1}{a+b\sin^2x}\right)dx$$
On dividing both the numerator and denominator by $\cos^2x$ and then letting $u$ substitution $u=\tan x$ we get $$I'(b) =\int_0^{\infty} \left(\frac {1}{a+(a-b)u^2}+ \frac {1}{a+(a+b)u^2}\right) du$$
And both of these simply evaluate using the fact that $$\int \frac {dx}{a^2+x^2}=\frac 1a \arctan \left(\frac xa\right)+ C$$
Hence $$I'(b) =\frac {\pi}{2\sqrt a}\left( \frac {1}{\sqrt {a-b}}+\frac {1}{\sqrt {a+b}}\right) $$
And now integrating again with respect to $b$ yields $$I(b)=\frac {\pi(\sqrt {a+b}-\sqrt {a-b})}{\sqrt a} +C$$
But using the fact that $I(0)=0$ we get $C=0$ and hence $$I(b)=\frac {\pi(\sqrt {a+b}-\sqrt {a-b})}{\sqrt a} $$
Q. E. D