I will do a couple of examples and see if you can do the rest.
Show the critical point is asymptotically stable if $q > 0$ and $p < 0,$ stable if $q > 0$ and $p = 0$ and unstable if $q < 0$ or $p > 0.$ $p$ and $q$ denote the trace and the determinant
of the matrix respectively.
- Stable if $q > 0$ and $p = 0$, we have: $\lambda_{1,2} = \pm i \sqrt{q} \rightarrow$ stable center.
- Unstable if $q < 0$ or $p > 0$, we have three cases here:
Case 1: If $D = (tr A)^2 - 4 \det A > 0$, and $tr(A) > 0$, it means that we have real eigenvalues and an unstable node.
Case 2: If $D = (tr A)^2 - 4 \det A > 0$, $\det A < 0 \rightarrow$ saddle.
Case 3: If $D = (tr A)^2 - 4 \det A < 0$, $\det A >0$, we have complex conjugate eigenvalues. If $tr(A) > 0$, we have an unstable spiral.
- Can you do the case $q > 0$ and $p < 0$? (You have a stable node or stable spiral.)
Now, can you relate these back to the critical point.
I would say a good start has been made on the right track.
Given the system
$\dot x = X(x, y) = -2x + x^2 + y^2, \tag 1$
$\dot y = Y(x, y) = y - xy, \tag 2$
I would find the critical points by first noting that
$y - xy = \dot y = 0 \tag 3$
implies
$y = xy; \tag 4$
thus if $y \ne 0, \tag 5$
we find
$x = 1; \tag 6$
now from (1) with $\dot x = 0$,
$-2(1) + 1^2 + y^2 = 0, \tag 7$
that is,
$y^2 - 1 = 0 \Longrightarrow y = \pm 1; \tag 8$
thus the critical points with $y \ne 0$ are $(1, -1)$ and $(1, 1)$. If $y = 0$ then (1) becomes
$x^2 - 2x = 0, \tag 9$
yielding
$x = 0, 2; \tag{10}$
therefore the critical points with $y = 0$ are $(0, 0)$ and $(2, 0)$.
So far, so good; our OP Khan Man and myself agree on the locations of the zeroes of the system (1)-(2).
I guess it's easiest to explain what it means to determine the stability and type of the critical points by proceeding with this example by way of illustration. We may find the Jacobean, or matrix of derivatives, of the system (1)-(2):
$J(x, y) = \begin{bmatrix} \dfrac{\partial X}{\partial x} & \dfrac{\partial X}{\partial y} \\ \dfrac{\partial Y}{\partial x} & \dfrac{\partial Y}{\partial y} \end{bmatrix} = \begin{bmatrix} 2x - 2 & 2y \\ -y & 1 - x \end{bmatrix}; \tag{11}$
at the critical point $(0, 0)$ this becomes
$J(0, 0) = \begin{bmatrix} -2 & 0 \\ 0 & 1 \end{bmatrix}; \tag{12}$
since the eigenvalues of $J(0, 0)$, $-2$ and $1$, are real and of opposite signs, $(0, 0)$ is an unstable saddle point; similarly, we may compute the Jacobean at the other zeroes of our system:
$J(2, 0) = \begin{bmatrix} 2 & 0 \\ 0 & -1 \end{bmatrix}; \tag{13}$
now the eigenvalues are $2$, $-1$, so we again find an unstable saddle; we proceed:
$J(1, -1) = \begin{bmatrix} 0 & -2 \\ 1 & 0 \end{bmatrix}; \tag{14}$
since $J(1, -1)$ is not diagonal we must resort to its characterisic polynomial to find the eigenvalues:
$\det (J(1, -1) - \mu I) = \det \left (\begin{bmatrix} -\mu & -2 \\ 1 & -\mu \end{bmatrix} \right ) = \mu^2 + 2 = 0; \tag{15}$
here the eigenvalues are $\mu = \pm \sqrt 2 i$; the point $(1, -1)$ is a center; since the eigenvalues have $0$ real part, we cannot determine the stability from the Jacobean alone; in such cases, a more detaied analysis is required; likewise,
$J(1, 1) = \begin{bmatrix} 0 & 2 \\ -1 & 0 \end{bmatrix}, \tag{16}$
and again we find that
$\mu^2 + 2 = 0, \; \mu = \pm \sqrt 2 i, \tag{17}$
so again we conclude that $(1, 1)$ is a center and the stability is indeterminate from this linearized analysis alone.
To summarize what has been illustrated here, the stability of a critical point is the property that trajectories initiated sufficiently close to it will remain so; generally this is associated with every eigenvalue of $J$ at the point in question having negative real part; if some eigenvalue has positive real part, then the point is unstable, and there will generally be a trajectory which moves away from the point, at least for awhile; our saddles exhibit this behavior, which is explained rigorously and in detail in the fine reference cited by user539887 in his comment on the question itself. There are no nodes or spiral points here, since they require all eiigenvalues to have non-zero real part of the same sign. A detailed explanation of all this may be found here.
Best Answer
The system is $y'=Ay+u$, where $A= \begin{bmatrix} 2 & 1 \\ -5 & 0 \end{bmatrix}$, and $u = \begin{bmatrix} 0 \\ 5 \end{bmatrix}$.
The critical points are where $y'=0$, this gives $y^* = -A^{-1} u = \begin{bmatrix} 1 \\ -2 \end{bmatrix}$.
The linearised system can be written as $(\upsilon+y^*)' = \upsilon' = A(\upsilon+y^*) +u = A \upsilon$.
Since the eigenvalues of $A$ are $1 \pm 2 i$, it is clear that the linearised system is unstable.