Consider the following 2D autonomous system of ODEs:
$$
\left\{
\begin{array}{ll}
\dot{x} = x^2 + 2y – x \\
\dot{y} = 3xy/2 – 3x^2 – y + 2x
\end{array}
\right.
$$
How can we prove the existence and uniqueness of a homoclinic orbit (i.e. a solution $X$ of the system
for which $\lim_{t \to \infty} X(t) = \lim_{t \to -\infty} X(t) = x_0,$ where $x_0$ is an equilibrium point of the system) for the system?
It is not hard to determine the equilibrium points: $(0,0), (-3, -6)$ and $(2/3, 1/9)$. The latter two are asymptotically stable equilibrium points, so there cannot be a homoclinic orbit for those points.
However, $(0,0)$ is a saddle equilibrium, so perhaps there is a homoclinic orbit here.
We can also see that the function $H(x,y) = – y^2 + x^2(1-x)$ is constant on the solutions of the system.
The set $H(x,y) = 0$ seems to be comprised of three solutions of the system, one of which is indeed a homoclinic orbit, so this proves existence.
However, how do we prove that this is unique? I don't really know how to approach this part.
Best Answer
Let me explain the idea from my previous answer with few more additional details.
As it was mentioned in the question, the system has a smooth first integral $H(\mathbf{x})$ — a (locally non-constant) function that is constant along the system's trajectories. The key observation is that if such system has a trajectory $\gamma(t)$ homoclinic to a saddle $p$, then $H(p) \equiv H(\gamma(t))$. To prove that we can pick a sequence of moments $t_i \rightarrow +\infty$. By continuity, since $\gamma(t_i) \rightarrow p$ and $H(\mathbf{x})$ is smooth, then $H(\gamma(t)) \rightarrow H(p)$. But since $H(\mathbf{x})$ is constant at any point of $\gamma(t)$, we have that $H(\gamma(t)) \equiv H(p)$.
This gives us the following method to find all homoclinic trajectories for a 2D system of differential equations. First, find all saddle equilibria. Second, find the level sets of $H(\mathbf{x})$ that contain these saddles. What we have proven before means that a homoclinic trajectory must lie in the same level set as the saddle, to which it is homoclinic. Just take a look at level sets after that and you can find homoclinic (and even heteroclinic) trajectories this way.