Consider the autonomous dynamical system,
$$\dot x = f(x)$$
An equilibrium $p$, ($f(p) = 0$), is unstable in the sense of Lyapunov means:
$\exists \epsilon > 0$ such that $\forall \delta > 0$, there exists some $x(0)$, whenever $\|x(0) – p\| < \delta$, $\|x(t) – p\| \geq \epsilon$ for some time $t$.
Since $x(t)$ only escapes the $\epsilon$ ball for some time $t$, but may come back in the future and even converge towards $p$, therefore can the equilibrium $p$ be simultaneously unstable and asymptotically stable?
I guess the correct definition of unstable should say: $\exists \epsilon > 0$ such that $\forall \delta > 0$, there exists some $x(0)$, whenever $\|x(0) – p\| < \delta$, there exists some $t_0$ such that $\|x(t) – p\| > \epsilon$ for ALL time $t \geq t_0$.
But this is not the negation of stable in the sense of Lyapunov (Recall stable in the sense of Lyapunov means $\|x(t) – p\| < \epsilon$ for all time $t$. Therefore the negation of this condition is $\|x(t) – p\| \geq \epsilon$ for some $t$. But this says nothing about the limiting behavior of $x(t)$.)
Best Answer
No, it cannot. By definition, an equilibrium point is asymptotically stable if it is both stable and attractive. Therefore, if the equilibrium point is unstable, it cannot be asymptotically stable.
However, the equilibrium point can be attractive without being stable. For instance, the equilibrium point $(0,0)$ of the system $$ \begin{array}{rcl} \dot{x}&=&x^2-y^2\\ \dot{y}&=&2xy \end{array} $$ exhibits such a property.
Your definition of unstable exclude attractivity, while the standard one allows for it. The definition of a stable equilibrium point $p$ in the sense of Lyapunov is
$$ \forall \epsilon>0,\exists \delta>0: ||x_0-p||\le\delta \Rightarrow \forall t\ge0:||x(t)-p||\le\epsilon. $$
In this regard, the definition of a unstable (or not-stable) equilibrium point $p$ in the sense of Lyapunov is given by the negation of the above statement:
$$ \exists \epsilon>0,\forall \delta>0: ||x_0-p||\le\delta \Rightarrow \exists t\ge0: ||x(t)-p||>\epsilon. $$