Hint: Split it into two integrals. One where $2y+2x=6$ is under $y=3$ and one where $2y=4\sqrt x$ is under $y=3$. Then find where $2y=4\sqrt x$ and $2y+2x=6$ intersect, and that's the bound between the two integrals.
If you don't want two integrals, you can also integrate with respect to $y$, where the height is the difference between $2y=4\sqrt x$ and $2y+2x=6$ and the bounds are from $2$ to $3$.
Setting $x=r \cos(\theta), y = r \sin(\theta)$ (the classical change of variables : polar $\to$ cartesian) in your equation gives
$$r^4(\underbrace{(\cos(\theta)^2+\sin(\theta)^2)^2}_{=1}-\underbrace{2\cos(\theta)^2.\sin(\theta)^2}_{\tfrac12 \sin(2 \theta)})=r^2 \underbrace{2 \cos(\theta).\sin(\theta)}_{\tfrac12 \sin(2 \theta)}.$$
Simplifying by $r^2$ and taking the square root, one gets :
$$r(\theta)=\sqrt{\dfrac{\sin(2\theta)}{1-\tfrac12\sin(2\theta)^2}} \ \ \
\text{if} \ \ \ \sin{2 \theta} \geq 0$$
The last condition means that the curve will exist if $0 \leq \theta \leq \pi/2$ (first quadrant) and/or $\pi \leq \theta \leq 3\pi/2$ (third quadrant). No part of the curve in the second and fourth quadrants.
It remains to integrate to obtain the area enclosed by the curve:
$$A=\frac12\int_0^{2 \pi}r(\theta)^2 d \theta$$
BUT, this integral is equal to $0$ because we turn once in the positive sense, once in the negative one. We have to take a serious look at the curve :
We are obliged, if we want to have the unsigned area of a "petal" to integrate from $0$ to $\pi/2$ (and afterwards double the result as a final answer). Up to you for the calculations (as you desire mainly hints).
Best Answer
Let's do an area preserving ($\det(T)=1$) change of coordinates, shall we:
We want a transformation which is something like $(x,y) \to (ax,by)$. If we had such a transformation, then, plugging in, we'd get something like: $b^2a^2x^2 + a^2b^2y^2=a^2b^2 \iff x^2+y^2=1$ which is just the equation of a circle.
So our matrix should be $\begin{bmatrix} a & 0 \\ 0 & b\end{bmatrix}$ because $$\begin{bmatrix} a & 0 \\ 0 & b\end{bmatrix}\begin{bmatrix} x \\ y \end{bmatrix} = \begin{bmatrix} ax \\ by \end{bmatrix}$$
Notice, though, that this is not an area-preserving transformation unless $ab$ happens to equal $1$. However, this one is:
$$\left(\frac 1{\sqrt{ab}}\begin{bmatrix} a & 0 \\ 0 & b\end{bmatrix}\right)\begin{bmatrix} x \\ y \end{bmatrix} = \frac 1{\sqrt{ab}}\begin{bmatrix} ax \\ by \end{bmatrix}$$
So doing the change of coordinates $(x,y) \to (\sqrt{\frac ab}x, \sqrt{\frac ba}y)$ won't change the area of your ellipse.
Plugging this in, we get $x^2 + y^2 = ab$. I assume you can find the area of this?