\begin{align}
x' &= -x^3 + x^5 + (x^4)(y^5)\\[.7em]
y' &= -8y^3 + y^5 – 10(y^4)(x^5)
\end{align}
$(0,0)$ is obviously a critical point of the system, and we are given that it is asymptotically stable, but have to show it.
I have tried to make a Lyapunov function $V(x,y) = ax^2 + cy^2$, with a,c > 0 but I am having trouble to prove that $\frac{d}{dt} V(x,y)$ is negative definite. I get some complicated polynomial I can't use logic to finalize. How can I change the Lyapunov to come up with a meaningful conclusion?
\begin{align}
\frac{d}{dt}V(x,y) = 2ax(-3x^2 + 5x^4 + 4x^3y^5) + 2cy(-24y^2+5y^4-40y^3x^5)\\[.7em]
\end{align}
Best Answer
For $$V(x,y)=\frac{1}{2}(x^2+y^2)$$ we have $$\dot{V}=-x^4-8y^4+x^6+y^6-9y^5x^5\\ \leq -(1-x^2-|xy|)x^4-8(1-(1/8)y^2-|xy|)y^4$$ Thus if we define the region (neighborhood of the origin) $$\Omega:=\left\{(x,y)|(x^2+|xy|<1)\textrm{ and }\frac{1}{8}y^2+|xy|<1\right\}$$ we have that $$\dot{V}(x,y)<0\qquad \forall (x,y)\in\Omega\neq 0$$ Now if we choose the level set $$\Omega_0:=\left\{(x,y)|V(x,y)<\frac{1}{4}\right\}\subseteq \Omega$$ we also have that $$\dot{V}(x,y)<0\qquad \forall (x,y)\in\Omega_0\neq 0$$ Therefore every solution that starts within $\Omega_0$ remains therein and asymptotically converges to the origin ($V$ is strictly decreasing).