The original question states
Let $g$ be a continuously differentiable function such that $g(0)=0$. Show that the solution $u \equiv 0$ of $u''+g(u)=0$ is not stable when $g'(0)<0$, and stable but not asymptotically stable when $g'(0)>0$
The first part is easy enough with Lyapunov linearization. However I am not able to prove the second part. I tried using Lyapunov functions but didn't find one.
I do think that the solutions whose value at 0 is close enough to 0 are all periodic. for example, $u''+u=0$ yield $\sin(x), \cos(x)$, while $u''+e^u-1=0$ and $u''+\sin(u)=0$ lead these and these solutions respectively. If this is true, it would solve the exercise, but I'm not sure if I can prove it.
Best Answer
The Lyapunov function is $$ V(u,u')= \int_0^u g(u)\, du+\frac12 (u')^2. $$ Indeed, $g(0)=0$, $g'(0)>0$ implies that for sufficiently small $u>0$, $g(u)$ is positive, and for sufficiently small $u<0$, $g(u)<0$, thus, in some neighborhood of $0$, $\int_0^u g(u)\, du>0$. Hence, $V(u,u')$ is positive definite. Obviously, $V(0,0)=0$ and $V$ is continuously differentiable.
The derivative $$ V'= \frac{\partial V}{\partial u} u'+\frac{\partial V}{\partial u'}u''= g(u)u'+u'(-g(u))=0, $$ thus, the origin is stable. Since $V(u,u')$ is the first integral of the system, it is not asymptotically stable.
The periodicity of the solutions is following from the fact that the level sets of Lyapunov functions are diffeomorphic to the sphere (see, for instance, this article).