Differentiate the first wrt $t$ to gain an expression for $\ddot y$:
$\dddot{x} = \frac{\omega_1^2}{2} \dot x + \omega_2 \ddot{y}$
Substitute $\ddot{y} = \frac{\omega_1^2}{2} y - \omega_2 \dot{x}$ to get:
$\dddot{x} = \frac{\omega_1^2}{2} \dot x + \frac{\omega_1^2 \omega_2}{2} y - \omega_2^2 \dot{x}$
Rearrange: $\frac{\omega_1^2 \omega_2}{2} y =\dddot{x} - \frac{\omega_1^2}{2} \dot x + \omega_2^2 \dot{x}$
Differentiate: $\frac{\omega_1^2 \omega_2}{2} \dot y =\ddddot{x} + \frac{2\omega_2^2 -\omega_1^2}{2} \ddot x$
Recall that $\ddot{x} = \frac{\omega_1^2}{2} x + \omega_2 \dot{y} \Rightarrow \omega_2 \dot{y}=\ddot{x} - \frac{\omega_1^2}{2} x$
Thus: $\frac{\omega_1^2}{2} \left (\ddot{x} - \frac{\omega_1^2}{2} x \right ) =\ddddot{x} + \frac{2\omega_2^2 -\omega_1^2}{2} \ddot x$
... which becomes $\ddddot{x} + \frac{2\omega_2^2 -\omega_1^2}{2} \ddot x - \frac{\omega_1^2}{2} \left (\ddot{x} - \frac{\omega_1^2}{2} x \right )=0 $
or $\ddddot{x} + \left(\omega_2^2 -\omega_1^2 \right ) \ddot x - \frac{\omega_1^4}{2}x =0 $
Auxiliary equation $\lambda^4+p\lambda^2-q=0$ where $p=\left(\omega_2^2 -\omega_1^2 \right )$ and $q=\omega_1^4$
$\lambda^2={-p+\sqrt{p^2+4q} \over 2}$ or $\lambda^2={-p-\sqrt{p^2+4q} \over 2}$
$\lambda_1=\sqrt{{-p+\sqrt{p^2+4q} \over 2}}$
$\lambda_2=-\sqrt{{-p+\sqrt{p^2+4q} \over 2}}$
$\lambda_3=\sqrt{{-p-\sqrt{p^2+4q} \over 2}}$
$\lambda_4=-\sqrt{{-p-\sqrt{p^2+4q} \over 2}}$
I would say a good start has been made on the right track.
Given the system
$\dot x = X(x, y) = -2x + x^2 + y^2, \tag 1$
$\dot y = Y(x, y) = y - xy, \tag 2$
I would find the critical points by first noting that
$y - xy = \dot y = 0 \tag 3$
implies
$y = xy; \tag 4$
thus if $y \ne 0, \tag 5$
we find
$x = 1; \tag 6$
now from (1) with $\dot x = 0$,
$-2(1) + 1^2 + y^2 = 0, \tag 7$
that is,
$y^2 - 1 = 0 \Longrightarrow y = \pm 1; \tag 8$
thus the critical points with $y \ne 0$ are $(1, -1)$ and $(1, 1)$. If $y = 0$ then (1) becomes
$x^2 - 2x = 0, \tag 9$
yielding
$x = 0, 2; \tag{10}$
therefore the critical points with $y = 0$ are $(0, 0)$ and $(2, 0)$.
So far, so good; our OP Khan Man and myself agree on the locations of the zeroes of the system (1)-(2).
I guess it's easiest to explain what it means to determine the stability and type of the critical points by proceeding with this example by way of illustration. We may find the Jacobean, or matrix of derivatives, of the system (1)-(2):
$J(x, y) = \begin{bmatrix} \dfrac{\partial X}{\partial x} & \dfrac{\partial X}{\partial y} \\ \dfrac{\partial Y}{\partial x} & \dfrac{\partial Y}{\partial y} \end{bmatrix} = \begin{bmatrix} 2x - 2 & 2y \\ -y & 1 - x \end{bmatrix}; \tag{11}$
at the critical point $(0, 0)$ this becomes
$J(0, 0) = \begin{bmatrix} -2 & 0 \\ 0 & 1 \end{bmatrix}; \tag{12}$
since the eigenvalues of $J(0, 0)$, $-2$ and $1$, are real and of opposite signs, $(0, 0)$ is an unstable saddle point; similarly, we may compute the Jacobean at the other zeroes of our system:
$J(2, 0) = \begin{bmatrix} 2 & 0 \\ 0 & -1 \end{bmatrix}; \tag{13}$
now the eigenvalues are $2$, $-1$, so we again find an unstable saddle; we proceed:
$J(1, -1) = \begin{bmatrix} 0 & -2 \\ 1 & 0 \end{bmatrix}; \tag{14}$
since $J(1, -1)$ is not diagonal we must resort to its characterisic polynomial to find the eigenvalues:
$\det (J(1, -1) - \mu I) = \det \left (\begin{bmatrix} -\mu & -2 \\ 1 & -\mu \end{bmatrix} \right ) = \mu^2 + 2 = 0; \tag{15}$
here the eigenvalues are $\mu = \pm \sqrt 2 i$; the point $(1, -1)$ is a center; since the eigenvalues have $0$ real part, we cannot determine the stability from the Jacobean alone; in such cases, a more detaied analysis is required; likewise,
$J(1, 1) = \begin{bmatrix} 0 & 2 \\ -1 & 0 \end{bmatrix}, \tag{16}$
and again we find that
$\mu^2 + 2 = 0, \; \mu = \pm \sqrt 2 i, \tag{17}$
so again we conclude that $(1, 1)$ is a center and the stability is indeterminate from this linearized analysis alone.
To summarize what has been illustrated here, the stability of a critical point is the property that trajectories initiated sufficiently close to it will remain so; generally this is associated with every eigenvalue of $J$ at the point in question having negative real part; if some eigenvalue has positive real part, then the point is unstable, and there will generally be a trajectory which moves away from the point, at least for awhile; our saddles exhibit this behavior, which is explained rigorously and in detail in the fine reference cited by user539887 in his comment on the question itself. There are no nodes or spiral points here, since they require all eiigenvalues to have non-zero real part of the same sign. A detailed explanation of all this may be found here.
Best Answer
The differential system is : $$ \begin{cases} \dot x_1=x_2 \\ \dot{x_2}=x_1^3+2x_1^2x_2-1 \end{cases} $$ You have a critical point at $(x_1,x_2)=(1,0)$. Change the variables: $$ \begin{cases} x_1=y_1+1 \\ {x_2}=y_2 \end{cases} $$
The jacobian matrix is: $$J=\pmatrix {0 & 1 \\ 3(y_1+1)^2+4(y_1+1)y_2 & 2(y_1+1)^2}$$ Now the critical point is at $(y_1,y_2)=(0,0)$ $$J=\pmatrix {0 & 1 \\ 3 & 2}$$ The linearized system now is: $$Y'=\pmatrix {0 & 1 \\ 3 & 2}Y$$ You compute the eigenvalues: $$P(\lambda)=\begin {vmatrix} -\lambda & 1 \\ 3 & 2-\lambda \end{vmatrix}$$ $$P(\lambda)=\lambda^2-2\lambda-3=0$$ $$\lambda_{1,2}=-1,3$$ So the critical point is a saddle point.