Another approach is to proceed using the Poincare-Bendixon theorem.
Our aim then is to show that the system contains only one unstable equilibrium point, and that every trajectory of the system remains in a closed bounded subset.
From your original problem, let
$$
f_1(x,y) = x - y - x^3 - \frac{3}{2}y^2{x} \nonumber \\
f_2(x,y) = x + y - x^2{y} - \frac{1}{2}y^3 \nonumber \\
$$
Then
$$
\dot{x} = f_1(x,y) \nonumber \\
\dot{y} =f_2(x,y) \nonumber \\
$$
The equilibrium point is $x=0$ and $y =0$
The Jacobian is
\begin{align}
J &=
\begin{bmatrix}
\frac{\partial{f_1}}{\partial x} & \frac{\partial{f_1}}{\partial y} \\
\frac{\partial{f_2}}{\partial y} & \frac{\partial{f_2}}{\partial y} \\
\end{bmatrix} \nonumber
_{\substack{x=0 \\ y=0}} \\
&=
\begin{bmatrix}
1 & -1 \\
1 & 1 \\
\end{bmatrix}
\end{align}
The eigenvalues are $1+i$ and $1-i$.
Hence the system is unable (unstable node or focus), which is the first criterion of the Poincare-Bendixon theorem.
Secondly, let us choose a closed bounded subset given by
$$
V(x,y)=x^2 + y^2 \le c \nonumber
$$
where c is a positive constant.
Therefore the chosed subset $V(x,y)$ is a circle of radius $\sqrt c$.
We only need to show that there exists a finte value of $c$ for which the vector field of $f(x,y)$ never leaves the set enclosed by $c$.
This is identical to the statement:
$$
\nabla {V(x,y)} \cdot f(x,y) \le 0 \nonumber
$$
Now,
$$
\nabla {V(x,y)} \cdot f(x,y) = \frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \nonumber
$$
Therefore
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \nonumber
$$
$$
= 2x\left[x-y-\left(x^2+\frac{3}{2}y^2\right)x\right] + 2y\left[x+y-\left(x^2+\frac{1}{2}y^2\right)y\right] \nonumber \\
= 2{\left(x^2+y^2\right)} -2x^2{\left(x^2+y^2+\frac{1}{2}y^2\right)}-2y^2{\left(x^2+y^2-\frac{1}{2}y^2\right)} \\
= 2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 + y^4-x^2y^2
$$
But
$$
y^4-x^2y^2 \le \left(x^2+y^2\right)^2
$$
Therefore
$$
2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 + y^4-x^2y^2 \le 2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 +\left(x^2+y^2\right)^2
= 2{\left(x^2+y^2\right)} - \left(x^2+y^2\right)^2
$$
This implies that
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \le 2{\left(x^2+y^2\right)} - \left(x^2+y^2\right)^2 = 2c - c^2 = (2-c)c \nonumber
$$
Finally, if we choose $c=2$, we are guaranteed that
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \le 0 \nonumber
$$
which satisfies the second criterion of the Poincare-Bendixon theorem.
Therefore, it can be concluded that a limit cycle exists.
Best Answer
(a) Consider the function $$ V=(x+y-1)^2. $$ Its derivative along the trajectories of the system is $$ \dot V= 2(x+y-1)(\dot x+\dot y)= 2(x+y-1)\left(1-x-\frac{2xy}{2+x}+ \frac{2xy}{2+x} -y\right) $$ $$ =2(x+y-1)(1-x-y)=-2V^2\le 0, $$ thus, according to the LaSalle's invariance principle, the $\omega$-limit set of the system is contained in $K$.
(b) The right part of the system is equal to zero iff $x=1$, $y=0$, thus, $(1,0)$ is an unique steady state. $\dot y$ is negative on $K$ except for the point $(1,0)$ (because $\frac{2x}{2+x}<1$ for any $x<2$), thus, $y$ is decreasing along the trajectories on $K$, thus, any solution that contained in $K$ moves toward the point $(1,0)$. It means that the $\omega$-limit set of the system contains the point $(1,0)$ only, i.e. for any initial point in $\mathbb D$ $\lim_{t\to\infty} (x(t),y(t))=(1,0)$.
In order to prove the global asymptotic stability of $(1,0)$ we need also to prove its (local) Lyapunov stability. It is given by the Lyapunov function $W(x,y)=V(x,y)+y^2$ since the derivative $$ \dot W(x,y)= -2V^2+2y^2\left( \frac{2x}{2+x} -1 \right) $$ is negative in some neighborhood of the equilibrium point $(1,0)$.