I would proceed by thus , let $y = [\sec (x)]^2 $
then
$$dy = 2 \cdot \sec(x) \cdot \sec(x) \cdot \tan(x) \cdot dx = 2 \cdot ( \sec (x))^2 \cdot \tan(x) \cdot dx $$
so,
$$
2 \tan^2(x) \sec^2 (x) dx = \sec(x) \cdot \tan(x) \cdot dy = y(y-1)^\frac{1}{2} \cdot dy
$$
since $$\sec(x) = y^{\frac{1}{2}}$$ and by considering
positive square roots only $\tan y = ( \sec^2(x) – 1)^{1/2} = (y – 1)^{1/2}$. Thus the substitution $y = \sec^2 x$ yields
$$
2 \int \tan^2 (x) \sec^3(x) dx = \int (y(y – 1) )^{1/2} dy
$$
and this later form can be reduced to the standard form $\int(z^2 – a^2)^{1/2} dz$ since
$$
y(y-1)=(y-(1/2))^2 – (1/2)^2 .
$$
What are the other ways to integrate this expression, except for the substitution
$\tan^2(x)^2=\sec (x)^2 -1$ which gives
$$
\sec^5(x) – \sec^3 (x)
$$
in the integrand which I quite don't like.
Best Answer
Let you want to solve $\int R\big(\sin(x),\cos(x)\big)dx$ and you know that $$R\big(\sin(x),-\cos(x)\big)\equiv -R\big(\sin(x),\cos(x)\big)$$ then you can take $\sin(x)=t$ for a good substitution. We have here $$\int \tan^2(x)\sec^3(x)dx=\int \frac{\sin^2(x)dx}{\cos^5(x)}$$ and we can see the above statement is true for the last integrand. By taking $\sin(x)=t$, we have $$\int\frac{t^2}{(1-t^2)^3}dt$$ which can be solve by fractions method.