Another approach is to proceed using the Poincare-Bendixon theorem.
Our aim then is to show that the system contains only one unstable equilibrium point, and that every trajectory of the system remains in a closed bounded subset.
From your original problem, let
$$
f_1(x,y) = x - y - x^3 - \frac{3}{2}y^2{x} \nonumber \\
f_2(x,y) = x + y - x^2{y} - \frac{1}{2}y^3 \nonumber \\
$$
Then
$$
\dot{x} = f_1(x,y) \nonumber \\
\dot{y} =f_2(x,y) \nonumber \\
$$
The equilibrium point is $x=0$ and $y =0$
The Jacobian is
\begin{align}
J &=
\begin{bmatrix}
\frac{\partial{f_1}}{\partial x} & \frac{\partial{f_1}}{\partial y} \\
\frac{\partial{f_2}}{\partial y} & \frac{\partial{f_2}}{\partial y} \\
\end{bmatrix} \nonumber
_{\substack{x=0 \\ y=0}} \\
&=
\begin{bmatrix}
1 & -1 \\
1 & 1 \\
\end{bmatrix}
\end{align}
The eigenvalues are $1+i$ and $1-i$.
Hence the system is unable (unstable node or focus), which is the first criterion of the Poincare-Bendixon theorem.
Secondly, let us choose a closed bounded subset given by
$$
V(x,y)=x^2 + y^2 \le c \nonumber
$$
where c is a positive constant.
Therefore the chosed subset $V(x,y)$ is a circle of radius $\sqrt c$.
We only need to show that there exists a finte value of $c$ for which the vector field of $f(x,y)$ never leaves the set enclosed by $c$.
This is identical to the statement:
$$
\nabla {V(x,y)} \cdot f(x,y) \le 0 \nonumber
$$
Now,
$$
\nabla {V(x,y)} \cdot f(x,y) = \frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \nonumber
$$
Therefore
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \nonumber
$$
$$
= 2x\left[x-y-\left(x^2+\frac{3}{2}y^2\right)x\right] + 2y\left[x+y-\left(x^2+\frac{1}{2}y^2\right)y\right] \nonumber \\
= 2{\left(x^2+y^2\right)} -2x^2{\left(x^2+y^2+\frac{1}{2}y^2\right)}-2y^2{\left(x^2+y^2-\frac{1}{2}y^2\right)} \\
= 2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 + y^4-x^2y^2
$$
But
$$
y^4-x^2y^2 \le \left(x^2+y^2\right)^2
$$
Therefore
$$
2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 + y^4-x^2y^2 \le 2{\left(x^2+y^2\right)} - 2\left(x^2+y^2\right)^2 +\left(x^2+y^2\right)^2
= 2{\left(x^2+y^2\right)} - \left(x^2+y^2\right)^2
$$
This implies that
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \le 2{\left(x^2+y^2\right)} - \left(x^2+y^2\right)^2 = 2c - c^2 = (2-c)c \nonumber
$$
Finally, if we choose $c=2$, we are guaranteed that
$$
\frac{\partial V}{\partial x}f_1(x,y) + \frac{\partial V}{\partial y}f_2(x,y) \le 0 \nonumber
$$
which satisfies the second criterion of the Poincare-Bendixon theorem.
Therefore, it can be concluded that a limit cycle exists.
Best Answer
Let me repeat, with suitable modifications, my answer to the OP's earlier question.
Assume that all solutions starting in $\mathbb{R}^2_{+} := \{\, (x, y) : x \ge 0,\ y \ge 0 \,\}$ are bounded for $t > 0$ (this would require a separate proof). It follows then that for any such solution, its domain contains $[0,\infty)$ and its $\omega$-limit set is compact and nonempty.
Let $L$ stand for the $ω$-limit set of some point, $(x_0,y_0)$, sufficiently close to the unstable focus $(1,3)$. By the Poincaré–Bendixson theorem, as there are finitely many equilibria, $L$ is either a limit cycle, or a heteroclinic cycle, (EDIT: or a homoclinic loop), or an equilibrium.
There are two equilibria, $(4,0)$ and $(1,3)$. The first of them is an unstable focus, so it cannot belong to any heteroclinic cycle (because a heteroclinic cycle (EDIT: or a homoclinic loop) must contain an equilibrium that is an $\omega$-limit point for some other point). Consequently, there are no heteroclinic cycles (EDIT: or homoclinic loops) at all.
So, $L$ is either a periodic orbit, or equals $\{(4,0)\}$. We proceed now to excluding the latter. The linearization of the vector field at $(4,0)$ has matrix $$ \begin{bmatrix} -1 & -\frac25 \\ 0 & 3 \end{bmatrix}. $$ There are two real eigenvalues, $-1$ and $3$, of opposite signs, so $(4,0)$ is a hyperbolic saddle. Its stable manifold is tangent to an eigenvector corresponding to $-1$, that is, to $(1,0)$. I claim that the stable manifold is the $x$-axis, minus $(4,0)$. Indeed, on the $x$-axis we have $\dot{x} = 4 - x$, $\dot{y} \equiv 0$, so for any $(x_1,0)$ we have $\omega((x_1,0)) = \{(4,0)\}$. Now, for a hyperbolic saddle its stable manifold is just the set of those points whose (unique) $\omega$-limit point is the saddle. Hence, if $L = \{(4,0)\}$ then the positive semitrajectory of $(x_0, y_0)$ must belong to the $x$-axis, which contradicts the uniqueness of the initial value problem (notice that $y_0 > 0$).
We have thus shown that $L$ is a periodic orbit.