Consider the system:
$\begin{cases}x' = e^{ay} – e^x \\ y' = a x^2 + (a-a^2) x + ay^2 e^{-y}\end{cases}$
Using Lyapunov first method we have, for $a \in ]-\infty,1[ \cup ]0,1[$ $p = (0,0)$ is unestable and for $a \in ]1,+\infty[$ is asymptotically stable.
For $a = 1$ I need to use Chetaev theorem to show that it is unestable and without indication I have to determine the stability when $a = 0$.
$a = 1$
In this case I looked for a Lyapunov function with separated variables but I get:
$\dot V(x,y) = V_1'(x) e^y – V_1'(x) e^x + x^2 V_2'(y) + y^2 e^{-y}V_2'(y)$
I don't see how to choose $V_1,V_2$ to make $\dot V(x,y) > 0$ in a neighbourhood of $p = (0,0)$
$a = 0$
This time we have the system $\begin{cases}x' = 1 – e^x \\ y' = 0\end{cases}$ and one solution is $(0,C)$ with $C \in \mathbb{R}$, so $p = (0,0)$ is not asymptotically stable. I tried to use Lyapunov's second theorem to have $\dot V = 0$ and show stability but it didn't work again.
Best Answer
The Chetaev function for $a=1$ is $V(x,y)=y$, since for any $(x,y):\; y>0$ we have $\dot V= x^2+y^2e^{-y}>0$.
For the case $a=0$, consider the Lyapunov function $V(x,y)=e^x-x-1+y^2$: $$ \dot V= e^x\dot x-\dot x= -(e^x-1)^2\le 0. $$