Consider the following system of equations
$$\begin{cases}
\dot x=y-x^2-x \\
\dot y=3x-x^2-y \\
\end{cases}
$$
Then, the equilibriums are $(0,0)$ and $(1,2)$.
Using linearization around $(1,2)$ one can obtain
$$
\begin{pmatrix}
\dot x \\
\dot y \\
\end{pmatrix}
=
\begin{pmatrix}
-3 & 1 \\
1 & -1 \\
\end{pmatrix}
\begin{pmatrix}
x \\
y \\
\end{pmatrix}
$$
The matrix has two distinct eigenvalues and both of them have negative real values, implying that the equilibria $(1,2)$ is asymptotically stable.
My problem is with $(0,0)$, where the linearization fails, as we have eigenvalues with positive real values
$$
\begin{pmatrix}
\dot x \\
\dot y \\
\end{pmatrix}
=
\begin{pmatrix}
-1 & 1 \\
3 & -1 \\
\end{pmatrix}
\begin{pmatrix}
x \\
y \\
\end{pmatrix}
$$
So, we need to construct a Lyapunov function, but the "usual" $V(x,y)=ax^2+by^2$ doesn't seem to work in this case.
Best Answer
Linear stability analysis comes to a useful conclusion as long as there are no eigenvalues with zero real part. If all eigenvalues have nonzero real part, the stability of the nonlinear system will match that of the linear system. Simply put, since you have two positive real eigenvalues, the fixed point is unstable in both systems.
The existence of a Lyapunov function implies stability, but it will be hard to show the existence of such a function, and even harder to show nonexistence, directly. Lyapunov functions should be used as a last resort, when linear stability analysis fails and you're pretty sure that the nonlinear system is stable, but you need to prove it.