Using $Mx-Ny$ method for reducing to exact differential equations (for which the criteria is satisfied here), it gets simplified to $\frac{3dx}{2x}+\frac{dy}{2y}-a\frac{ydx+xdy}{x^2}=0$. This is where I get stuck at having to find integral of $\frac{ydx+xdy}{x^2}$. Here, $Mx-Ny=2x^2y$.
Now I know this can be solved through various other methods. But I am trying to figure out if I can solve using $Mx-Ny$ method. But I have no idea of how to solve $\frac{ydx+xdy}{x^2}$ if it is even possible?
Edit: Realizing from @Oscar Lanzi's answer that $\frac{ydx+xdy}{x^2}$ is not integrable. Is there any other rearrangement of $\frac{(3xy-2ay^2)dx+(x^2-2axy)dy}{2x^2y}=0$ that is easily integrable?
Answer: $x^2y(x-ay)=c$
Best Answer
Your problem isn't stated correctly. If you reverse engineer the solution you find that its $M$ is $\frac{\partial}{\partial x} x^2 y(x-ay)=2xy(x-ay)+x^2 y = 3x^2y - 2axy^2$ and its $N$ is $\frac{\partial}{\partial y} x^2 y(x-ay)=x^2(x-ay)-ax^2y=x^3-2ax^2 y$. Thus, apparently the writer of the question divided the exact equation:
$$(3x^2 y - 2axy^2) dx + (x^3-2ax^2 y) dy = 0$$
by $x$ which results in
$$(3xy - 2ay^2) dx + (x^2-2axy) dy = 0.$$
This problem is not in the correct form to use the "Mx-Ny method" since for instance $3xy-2ay^2$ is not of the form $yf(xy)$. Instead if you try to work from scratch you arrive at the PDE
$$\mu_y (3xy-2ay^2) + \mu(3x-4ay) = \mu_x (x^2-2axy) + \mu(2x-2ay).$$
This simplifies to
$$\mu_y(3xy-2ay^2) + \mu(x-2ay) = \mu_x (x^2-2axy).$$
You see now that the coefficient on $\mu$ and the coefficient on $\mu_x$ only differ by a factor which is a function of $x$, which means you can assume $\mu_y=0$ and get
$$\mu = x \mu_x.$$
Thus $\mu=x$ is an admissible integrating factor (as we'd expect from how the reverse engineering went).