For these systems, you may have to pull out several tools to figure them out.
For the first one, you guessed at the energy function, but there is no known way to derive those (for some physical systems, you can, but no general way).
What other tools might we use?
- Find the critical points - for some pathological problems, this in itself can be very difficult - like some of your examples. For all of your examples, one of the critical points is $(0,0)$
- Find the Jacobian matrix and evaluate the eigenvalues at the critical points - sometimes this may not work.
- Plot the phase portrait and look at the qualitative behavior.
For the first problem, if we find the Jacobian matrix, evaluate it at the critical point $(0,0)$, we find the the eigenvalues are $-2$ and $0$, so what can you tell about the stability from that? You can try this approach for the others and see if it bears fruit. Note, for all of these, there is more than a single critical point and there are some strange behaviors going on, but that is to be expected given these systems.
Here are the four phase portraits for these systems (note, if you expand out the ranges, you can see wild behaviors).
Update
We are given:
$$ f(x, y) = x'= e^{x+2y}-\cos 3x\\ g(x,y) = y'= 2\sqrt{1+2x}-2e^y$$
It is clear that $(0,0)$ is a critical point.
Lets find the Jacobian matrix of this system, evaluate it at the critical point, determine the type of critical point and validate this behavior using the phase portrait. The Jacobian matrix is given by:
$$\displaystyle J(x,y) = \begin{bmatrix} \frac{\partial f}{\partial x} & \frac{\partial f}{\partial y}\\\frac{\partial g}{\partial x} & \frac{\partial g}{\partial y}\end{bmatrix} = \begin{bmatrix} e^{x+2y} + 3 \sin 3x & 2e^{x+2y} \\ \dfrac{2}{\sqrt{x^2+1}} & -2e^{y} \end{bmatrix}$$
Now, evaluated at the critical point $(0,0)$, we have:
$$J(0,0) = \begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}$$
The eigenvalues of this system are $2$ and $-3$, thus we have two real, distinct and opposite sign $\rightarrow$ a saddle point.
If you look at the coordinates $(0,0)$ of the third phase portrait, do you see the saddle point? What can you tell about the stability of a saddle point?
The phase portrait lets us see this qualitative behavior without having to do all that work.
This system can be represented as
$$
U_n = M U_{n-1}
$$
with $U_n = (n_n, y_n)^{\dagger}$ and $M = \left(\begin{array}{cc}0 & 1\\ \frac 1a & -\frac 1a\end{array}\right)$
but $M = T^{-1}\Lambda T$ then
$$
U_n = T^{-1}\Lambda T U_{n-1}\Rightarrow TU_n = \Lambda T U_{n-1}
$$
calling now $V_n = TU_n$ we have
$$
V_{n+1} = \Lambda V_n
$$
Here $\Lambda = \left(\begin{array}{cc}-\frac 12\left(\frac{1+\sqrt{1+4a}}{2a}\right) & 0\\ 0 & \frac 12\left(\frac{-1+\sqrt{1+4a}}{2a}\right)\end{array}\right)$ the diagonal eigenvalues matrix. Those eigenvalues have opposite signs characterizing a saddle point.
$$
V_n = \Lambda^n V_0
$$
Attached a plot showing in blue the two eigenvalues as well as the stability region when both eigenvalues are between the black limits $-1 < \lambda_1(a),\lambda_2(a) < 1$
Best Answer
Your answer is incorrect.
The fact that the eigenvalues are $0$ and $-1$ does not tell you whether the critical point is stable or unstable: it could be either. You need more information.
For example, $\dot{x} = x^2,\; \dot{y} = -y$ also has eigenvalues $0$ and $-1$ but has an unstable critical point.