Finding an integrating factor can be a genuine mathematical art.
However, certain differential forms can remind us of differentiation
techniques that may aid in the solution of the equation at hand. For
example, seeing $x\ dy + y\ dx$ reminds us of the product rule, as in
$d(xy) = x\ dx + y\ dy$ [sic], and $x\ dy – y\ dx$ might remind us of the
quotient rule, $d(x/y) = \frac{y\ dx – x\ dy}{y^2}$. In the equation
$x\ dy + y\ dx + 3xy^2\ dy = 0$, we are again reminded of the product
rule. In fact, if you multiply the equation by $\frac{1}{xy}$, then
$\frac{x\ dy\ +\ y\ dx}{xy} + 3y\ dy = 0 \implies d(ln(xy)) + 3y\ dy = 0 \implies ln(xy) + \frac{3}{2}y^2 = C$. Use these ideas to find a
general solution for the differential equation $$(x^2 – y^2)(x\ dy + y\ dx) = 2xy(x\ dy – y\ dx)$$
I'm not really sure where to start with this. I can rewrite the equation as $(x^2 – y^2)\ d(xy) = 2x^3y\ d(y/x)$ and solve for one of the differentials, but not in a way where I can integrate the RHS. I noticed that $x\ dy + y\ dx$ is cleanly expressible as a dot product, and since $d\vec r$ = $\begin{bmatrix}dx \\ dy\end{bmatrix}$, integrating $d(xy)$ could be viewed as calculating work, where $\vec f(x,\ y) = \begin{bmatrix}y \\ x\end{bmatrix}$. However, there doesn't appear to be a curve, so I can't set integration bounds or use Green's Theorem. Multiplying everything out didn't spark any ideas either.
Best Answer
Starting from your original differential equation of
$$(x^2 - y^2)(x\ dy + y\ dx) = 2xy(x\ dy - y\ dx) \tag{1}\label{eq1}$$
divide both sides by $x^2$ to get
$$\left(1 - \left(\frac{y}{x}\right)^2\right)(x\ dy + y\ dx) = 2xy\left(\frac{x\ dy - y\ dx}{x^2}\right) \tag{2}\label{eq2}$$
Using the suggestions for the differentials, we get
$$\left(1 - \left(\frac{y}{x}\right)^2\right)d\left(xy\right) = 2xy\left(d\left(\frac{y}{x}\right)\right) \tag{3}\label{eq3}$$
We now have a separable equation in $xy$ and $\frac{y}{x}$. Making the appropriate adjustments, i.e., dividing both sides by $xy\left(1 - \left(\frac{y}{x}\right)^2\right)$, gives
$$\frac{d\left(xy\right)}{xy} = \frac{2d\left(\frac{y}{x}\right)}{1 - \left(\frac{y}{x}\right)^2} \tag{4}\label{eq4}$$
You can now integrate both sides appropriately and simplify the results. I assume you can finish the remaining calculations yourself.