If I understand your question correctly, you are looking for an example where a Lyapunov candidate function proves the stability of a nonlinear system inside a region, while the system is still stable outside that region too.
Well, it is safe to say that most nonlinear systems and Lyapunov candidate functions are like this. Just search for the terms region of attraction or basin of attraction to see hundreds of results and papers on this topic.
Consider e.g. the Vanderpol oscillator:
$$\frac{d^2\theta}{dt^2}-\mu(1-\theta^2)\frac{d\theta}{dt}+\theta=0$$
which is converted to the standard form of
$$\dot x_1=x_2\\ \dot x_2=\mu(1-x_1^2)x_2-x_1$$
with $x=[x_1\;x_2]^T$ as the state vector. The trivial equilibrium point of this system is stable for $\mu<0$. This can be verified by setting the Lyapunov candidate as
$$V(x)=\frac 12x^Tx=\frac 12(x_1^2+x_2^2)$$
which results in
$$\dot V=x_1\dot x_1+x_2\dot x_2=\mu(1-x_1^2)x_2^2$$
As you see, the gradient of $V$ is negative when $\mu<0$ and $|x_1|<1$. In other words, the basin of attraction of this Lyapunov candidate is the region where $|x_1|<1$. But as you see in the below phase portrait (where $\mu=-1$), there are points outside that region which belong to the stable area of the system.
These are multiple question, I will try to answer some of them.
I assume that $a, b > 0$.
An equilibrium is isolated if there does not exist a small "ball" around it such that it is the only equilibrium in that ball. For example:
$$
\dot{x_1} = 0
$$
has no isolated equilibrium because no matter how close you look at an equilibrium, there are always infinitely many other equilibrium points close by. Similiar, the system
$$
\begin{align}
\dot{x}_1 &= x_1 - x_2 \\
\dot{x}_2 &= x_1 - x_2 \\
\end{align}
$$
has no isolated equilibrium: For example $x_1 = x_2 = 1$ is is an equilibrium, but so is $x_1 = x_2 = 1.1$ and $x_1 = x_2 = 1.01$ and $x_1 = x_2 = 1.00000001$ and so on. No matter how close you look, there are always "more" equilibrium points.
So to your example, you have:
$$
\begin{align}
\dot{x}_1 &= x_2 + x_3 \\
\dot{x}_2 &= -a \sin(x_1) - b x_2 \\
\dot{x}_3 &= -a \sin(x_1) + x_3
\end{align}
$$
To get the equilibrium points you set $\dot{x} = 0$ so you need $x_2 = -x_3$ from the first equation. Then you get
$$
\begin{align}
\dot{x}_2 &= \dot{x}_3 = 0 \\
\implies -a \sin(x_1) - b x_2 &= -a \sin(x_1) + x_3 \\
\implies -b x_2 &= x_3 \\
\implies -b x_2 &= -x_2
\end{align}
$$
We assumed that $b > 0$ and if $b \neq 1$ the last equation can only be true if $x_2 = 0$ and then $x_3 = 0$. If you insert that you get
$$
-a \sin(x_1) = 0
$$
so the equilbrium points are $x_1 = \sin(k \pi), k \in \mathbb{Z}, x_2 = 0, x_3 = 0$. Although that means you have infinitly many equilibrium points they are all isolated because the roots of $\sin(x_1) = 0$ are isolated. Conclusion: You can use Lyapunov method.
If $b = 1$ the system has no isolated equilibrium points.
You can use indirect method but it will only tell you local results and sometimes you can't use it (if you get zero eigenvalues).
You have the function
$$
V(x) = \frac{1}{2} x^T x + a(1 - \cos(x_1))
$$
but it is not a Lyapunov function for your system:
$$
\dot{V}(x) = - b x_2^2 + x_1 x_2 - x_3^2 + x_1 x_3
$$
If $x_3 = 0$ and $x_1 = 2 b x_2$ you get $\dot{V}(x) = b x_2^2$ and that is positive no matter how small you make $x_2$. So this function can't be used.
Best Answer
No, but there are extensions that allow you to do so.
You can look up input to state stability for example. But you will need some assumptions for $u$, for example that the input is bounded.
For example, the system
$$ \dot{x} = -x^3 + xu $$
is input to state stable if $|u| < C$ for any finite positive $C$ because as $x$ gets large the stabilizing $-x^3$ "beats" the $xu$.
But the system
$$ \dot{x} = -x^3 + x^3u $$
is not input to state stable for any bounded input, because with the input $u = 2$ you get
$$ \dot{x} = x^3 $$
which is unstable.
This is of course not very rigorous but there exists a rigorous theory and a lot of literature for this topic. However it gets quickly much more complicated if you have more complicated systems.