Consider system:
$\dot x = ax – {e^{\tanh \left( k \right)}}x$
$\dot k = – \frac{1}{{{{{\mathop{\rm sech}\nolimits} }^2}\left( k \right)}}\frac{1}{{{e^{\tanh \left( k \right)}}}}a{x^2}$
where, $a>e=2.71828\dots$ Choose a Lyapunov function as
$V(x,k) = \frac{1}{2}{x^2} + {e^{{\mathop{\rm tanh}\nolimits} \left( k \right)}}$
Its time derivative is
$\begin{aligned} \dot{V} &=x\left(a x-e^{\tanh ({k})} x\right)+e^{\operatorname{tanh}({k})} \operatorname{sech}^{2}({k}) \dot{{k}} \\ &=a x^{2}-e^{\tanh ({k})} x^{2}-a x^{2} \\ &=-e^{\tanh ({k})} x^{2} \leq 0 \end{aligned}$
This implies the asymptotic stability of $x$. However, since $a$ is larger than ${e^{\tanh \left( k \right)}}$, this system is clearly unstable. To be more specific, consider another "Lyapunov" function (clearly this should not be called Lyapunov function, see answer for clearification):
$U(x,k)=\frac{1}{2}x^2$
$\begin{aligned} \dot U &=x\left(a x-e^{\tanh ({k})} x\right) \\ &=\left(a-e^{\tanh ({k})}\right) x^{2} \geq 0 \end{aligned}$
According to Chetaev instability theorem, $x=0$ is an unstable equilibrium.
Why comes this contradiction?
Best Answer
Your Lyapunov function is invalid. You want to analyse the equilibrium $x = 0$, but your Lyapunov function does not have an isolated minimum there.
For example, $V(0,0) = \frac{1}{2}0^2 + e^{\tanh(0)} = 0 + 1 = 1$.
Now, for $x = 0$, $k < 0$ you will have $V(0, k) < V(0, 0)$.
Note 1: Your function $U$ is usually called Chetaev function rather than Lyapunov function. By definition, your $U$ is not a Lyapunov function as it is only positive semi-definite.
Edit: Expanded the answer for better clarity.
Note 2: One thing to note first is that your system has an equilibrium set with $x = 0$ and $k \in \mathbb{R}$ arbitrary. So, stability cannot be checked with the direct method, as it works only for isolated equilibria, not for a continuum of equilibria.
Note 3: To analyze stability of an isolated equilibrium, your Lyapunov function should have that point as an isolated minimum. So even if this system would have an isolated equilibrium, the analysis would be invalid because this $V$ does not have an isolated minimum at all.