Considering a potential energy of $U$, and a displacement of $x$, the force is given by
$F=-\frac{\partial U}{\partial x}$.
Since equilibrium is defined as the point at which $F=0$, we can express this as $\frac{\partial U}{\partial x}=0$. This is clear to see on the following graph;
It is also clear that some equilibria are stable and some are not; given a small displacement at $x_2$ the system will return to equilibrium, whereas this would not happen at $x_3$. Hence, we can say that for $\frac{\partial ^2U}{\partial ^2x}>0$ the equilibrium is stable, whereas for $\frac{\partial ^2U}{\partial ^2x}<0$ the equilibrium is unstable. Is there a general solution to this case, or does each have to be considered individually?
What is not clear to me is the case where $\frac{\partial ^2U}{\partial ^2x}=0$. Does this simply mean that the equilibrium is stable given a displacement in one direction and not the other, or is it more complicated – for example if a particle were to oscillate about a stable equilibrium point, its motion would be dampened until it were at rest, but this would not be possible at a point where $\frac{\partial ^2U}{\partial ^2x}=0$; if the particle were to move to the side where $\frac{\partial ^2U}{\partial ^2x}<0$, it would not return to the equilibrium point. Is there a general solution to this case, or does each case have to be considered by inspection?
Best Answer
Consider the following potentials:
\begin{align} U(x) &= x^4 \\ U(x) &= x^6 - x^4 \\ U(x) &= x^4 + x^3 \end{align}
All three of these potentials have an equilibrium point at $x = 0$. All three of these potentials are such that the second derivative of $U(x)$ at this equilibrium point is zero. However, you should convince yourself (perhaps by plotting these potentials) that in the first case the equilibrium is stable, in the second case it is unstable, and in the third case the equilibrium is, as you put it, "stable in one direction but unstable in the other".
The moral is: knowing only that the second derivative is zero tells us nothing about stability. We need to look at higher derivatives if we want to know more.