For these systems, you may have to pull out several tools to figure them out.
For the first one, you guessed at the energy function, but there is no known way to derive those (for some physical systems, you can, but no general way).
What other tools might we use?
- Find the critical points - for some pathological problems, this in itself can be very difficult - like some of your examples. For all of your examples, one of the critical points is $(0,0)$
- Find the Jacobian matrix and evaluate the eigenvalues at the critical points - sometimes this may not work.
- Plot the phase portrait and look at the qualitative behavior.
For the first problem, if we find the Jacobian matrix, evaluate it at the critical point $(0,0)$, we find the the eigenvalues are $-2$ and $0$, so what can you tell about the stability from that? You can try this approach for the others and see if it bears fruit. Note, for all of these, there is more than a single critical point and there are some strange behaviors going on, but that is to be expected given these systems.
Here are the four phase portraits for these systems (note, if you expand out the ranges, you can see wild behaviors).
Update
We are given:
$$ f(x, y) = x'= e^{x+2y}-\cos 3x\\ g(x,y) = y'= 2\sqrt{1+2x}-2e^y$$
It is clear that $(0,0)$ is a critical point.
Lets find the Jacobian matrix of this system, evaluate it at the critical point, determine the type of critical point and validate this behavior using the phase portrait. The Jacobian matrix is given by:
$$\displaystyle J(x,y) = \begin{bmatrix} \frac{\partial f}{\partial x} & \frac{\partial f}{\partial y}\\\frac{\partial g}{\partial x} & \frac{\partial g}{\partial y}\end{bmatrix} = \begin{bmatrix} e^{x+2y} + 3 \sin 3x & 2e^{x+2y} \\ \dfrac{2}{\sqrt{x^2+1}} & -2e^{y} \end{bmatrix}$$
Now, evaluated at the critical point $(0,0)$, we have:
$$J(0,0) = \begin{bmatrix} 1 & 2 \\ 2 & -2 \end{bmatrix}$$
The eigenvalues of this system are $2$ and $-3$, thus we have two real, distinct and opposite sign $\rightarrow$ a saddle point.
If you look at the coordinates $(0,0)$ of the third phase portrait, do you see the saddle point? What can you tell about the stability of a saddle point?
The phase portrait lets us see this qualitative behavior without having to do all that work.
The computation of the equilibrium points is not correct. Since you do not give any assumptions on the parameters, let us suppose that $\alpha,\beta,\gamma$ are non zero. you need to solve simultaneously
\begin{equation}
\alpha x-\beta xy=0; \qquad \beta xy-\gamma y=0.
\end{equation}
The point $(0,0)$ is clearly an equilibrium point. Observe that $x=0$ implies $\gamma y=0$ and that $y=0$ implies $\alpha x=0$. Thus, your last two equilibrium points are not correct.
Now, suppose $(x,y)$ is not the origin. Then $(x,y)=(\gamma/\beta,\alpha/\beta)$ is the other equilibrium point.
The Jacobian is the matrix
\begin{equation}
J=\begin{bmatrix}\alpha-\beta y & -\beta x\\
\beta y & \beta x-\gamma
\end{bmatrix},
\end{equation}
then
\begin{equation}
J|_{(0,0)}=\begin{bmatrix}\alpha& 0\\
0 & -\gamma
\end{bmatrix}; \qquad J|_{(\gamma/\beta,\alpha/\beta)}=\begin{bmatrix}0 & -\gamma\\
\alpha & 0
\end{bmatrix}
\end{equation}
You could construct the following table, depending on the eigenvalues of $J$ at each point. Let $p_1=(0,0)$, $p_2=(\gamma/\beta,\alpha/\beta)$
\begin{array}{|c|c|c|c|}
\hline
\alpha & \beta & \gamma & p_1 & p_2 \\ \hline
+ & + & + & saddle & center \\ \hline
+ & + & - & source & saddle \\ \hline
+ & - & + & saddle & center \\ \hline
+ & - & - & source & saddle \\ \hline
- & + & + & sink & saddle \\ \hline
- & + & - & saddle & center \\ \hline
- & - & + & sink & saddle \\ \hline
- & - & - & saddle & center \\ \hline
\end{array}
And then, for example, have the following phase portraits (at least for the first 2 cases, you could try to do the rest)
Best Answer
Your original system can be rewritten as:
$$\begin{bmatrix} \dot{y}_1 \\ \dot{y}_2 \end{bmatrix}=\begin{bmatrix} -2 & 1 \\ -1 & 0 \end{bmatrix} \begin{bmatrix} y_1 \\ y_2 \end{bmatrix}$$
Where $y_1=y_1(t)$, $y_2=y_2(t)$ are the functions to be found.
Let's consider a general system of linear ODEs with constant coefficients:
$$\dot{\vec{y}}=\hat{A} \vec{y}$$
Indeed, a partial solution to such a system can be expressed through the eigenvalues and eigenvectors of a square matrix $\hat{A}$ because:
$$\hat{A} \vec{a}_j=v_j \vec{a}_j$$
$$\vec{y}_j=\vec{a}_j e^{v_j t}$$
Where $\vec{a}_j$ and $v_j$ are eigenvectors and eigenvalues respectively. $j=1,...,n$ where $n$
Using this, we can write:
$$\dot{\vec{y}}_j=v_j\vec{a}_j e^{v_j t}=\hat{A} \vec{a}_j e^{v_j t}=\hat{A}\vec{y}_j$$
We can write the general solution as a combination of partial solutions for all eigenvalues and eigenvectors:
$$\vec{y}(t)=\sum_j^n C_j \vec{a}_j e^{v_j t}$$
For some arbitrary constants $C_j$.
Now eigenvalues for small $n$ are indeed found by solving the equation:
$$\det (\hat{A}-v\hat{I})=0$$
In this case the equation becomes:
$$(-2-v)(-v)+1=0 \\ v^2+2v+1=0 \\ (v+1)^2=0 \\ v=-1$$
We get a degenerate case, when the two eigenvalues are the same (repeated eigenvalues).
Because of that, for the general solution we need to take the following two vectors:
$$\vec{y}_1=\vec{a} e^{vt} \\ \vec{y}_2=(\vec{b}+\vec{a} t)e^{vt}$$
Where $\vec{b}$ is the solution of:
$$(\vec{A}-v \vec{I}) \vec{b}=\vec{a}$$
Why can we do that I leave for you to find in your lectures/textbooks.
This leaves us to find the eigenvector $\vec{a}=(a_1,a_2)^T$ for $v=-1$. We need to solve:
$$-2a_1+a_2=-a_1 \\ -a_1=-a_2$$
We get $a_1=c$, $a_2=c$ for some arbitrary constant $c$. Since we are searching for a partial solution we can take $c=1$.
The vector $\vec{b}$ is found from:
$$(-2+1) b_1+b_2=1 \\ -b_1+b_2=1$$
Again, we get multiple solutions, so I think we can choose whatever we want, like:
$$b_1=0 \\ b_2=1$$
The general solution can be expressed as:
For arbitrary constants $C_1,C_2$. We can check if it works by substitution:
$$\dot{\vec{y}}(t)=(C_2-C_1) \begin{bmatrix} 1 \\ 1 \end{bmatrix} e^{-t}-C_2 \left(\begin{bmatrix} 0 \\ 1 \end{bmatrix}+\begin{bmatrix} 1 \\ 1 \end{bmatrix} t \right) e^{-t}$$
$$\hat{A} \vec{y}=-C_1 \begin{bmatrix} 1 \\ 1 \end{bmatrix} e^{-t}+C_2 \left(\begin{bmatrix} 1 \\ 0 \end{bmatrix}-\begin{bmatrix} 1 \\ 1 \end{bmatrix} t \right) e^{-t}$$
The two expressions after simplification are equal, so our solution is correct.