One needs to show why solutions for the system $$x'=\left[\begin{matrix} 0&-1&0 \\ 0&-2&0 \\ -1&2&-1\end{matrix}\right]x$$ are Lypunov or asymptotically stable/unstable. Most probably we would need to check eigenvalues real part to determine stability. However $\lambda_1=0, \lambda_2=-1, \lambda_3=-2$, which makes the whole process a little more difficult, especially when it comes to check the former type of stability (eigenvalues are nonpositive and that implies solutions are Lyapunov stable).
But how do I know if they are asymptotically stable? Shoould I find strictly decreasing Lyapunov function?
Best Answer
Zero eigenvalue means $x'$ is zero on some line. (Specifically, the line $x_1+x_3=0=x_2$.) What happens to solutions that begin on that line? They stay where they started. So much for asymptotic stability.
In general, an asymptotically stable equilibrium must be an isolated equilibrium: since nearby points are drawn to it, they can't be equilibria too.