Let me expand on Art's observation a bit.
In the equation, $b$ should be replaced by $by$, it's the restoring force of user1631, too. There is no "vector term" in the matrix equation anymore: the equation or set of equations is linear. The matrix is
$$ \pmatrix{0&1\\ b & -a} $$
The eigenvalues are solutions to
$$ (\lambda-0)(\lambda+a) + b = 0$$
i.e.
$$\lambda = \frac{ -a\pm \sqrt{a^2 - 4b } }{2}$$
where we have somewhat permuted letters relatively to the usual formula fo the quadratic equation. For $a^2-4b\gt 0$, the eigenvalues are real and negative, implying damped motion: the oscillator will never swing to the opposite sign. For $a^2-4b\lt 0 $, the eigenvalues are complex, complex conjugate to each other, and having a negative real part. It means that the motion is still damped by it's also oscillating. At any rate, due to the damping, the behavior is stable for all cases.
As a check, for your linear stability matrix I get:
$$A = {\left. {\left( {\matrix{
{3{x^2} + a} & 1 \cr
1 & { - 1} \cr
} } \right)} \right|_{(x* = 0,y* = 0)}} = \left( {\matrix{
a & 1 \cr
1 & { - 1} \cr
} } \right)$$
which is presumably what you have as well. By the way, did you switch notation from, $a \to \alpha $?
So the determinant, $\Delta $, and trace, $\tau $, are given by $-(a + 1)$ and, $a - 1$, respectively.
As you are aware, but just so we are on the same page, the classification of a given fixed point is determined by the values of $\Delta$ and $\tau$, for a given value of $a$. As you mention, with $a<-1$, $\Delta>0$, so the fixed point is a stable node because, ${\tau ^2} - 4\Delta> 0$ with $\tau<0$, and for $a>-1$, you have a saddle node since the $\Delta<0$.
For $a=-1$ (assuming that by $\alpha$ you meant $a$) we have $\Delta = 0$, which signals that you are at a bifurcation point, meaning that the topological structure of the phase diagram is transitioning at this value of $a$. In your case the transition is from a stable node to a saddle node.
You can convince yourself of this visually in Mathematica by plotting something like:
PlotVectorField[{a x + y + x^3, x - y}, {x, -L , L}, {y, -L, L},
Axes -> True]
For suitably chosen values of the parameters, $a$ and $L$.
There are conventions on how to classify the different types of bifurcations that can occur, e.g., saddle-node, transcritical, pitchfork, critical, Hopf, transcritical, etc. See e.g., Strogatz for more discussion on the classifications, and in particular, to discover the type of bifurcation point you have in your system.
Hope this helps.
Best Answer
Let us look at a one-dimensional example:
Recall that $f(x) = \dot{x}$, so $f$ encodes the time evolution of $x$. If $f < 0$, then $x$ will move to the left. If $f > 0$, then $x$ will move to the right. If $f = 0$, $x$ will not move at all, this is why $f(x_0) = 0$ is the equilibrium condition.
Now, look what happens if you perturb the equilibria $x_i$ slightly to the right: If $f'(x_i) < 0$, then, in a small surrounding of $x_i$, $f(x) < 0$ for $x > x_0$ in that surrounding, i.e. to the right of a equilibrium with $f'(x_i) < 0$, $x$ will move to the left - returning to equilibrium! Conversely, if $f'(x_i) > 0$, then $f(x) > 0$ for all $x$ to the right in the small surrounding, meaning that, after a small nudge to the right, $x$ will move even further right, leaving equilibrium!
The same line of reasoning can be applied to perturbations to the left, altogether showing that, if $f'(x_i) < 0$, then a small perturbation around $x_i$ will always move back into $x_i$, and if $f'(x_i) > 0$, then a small perturbation will become bigger and bigger, moving away from $x_i$.
The multi-dimensional case is less graphic, but the intuition is the same - negative eigenvalues of the Jacobian mean that the time evolution points back into equilibrium, positive eigenvalues mean that it points away from equilibrium.