The other answers here are in the spirit of what you can do, but allow me to elaborate a little more.
To understand if the trajectory of the movement under a potential $V $ is stable or not you have to understand what this stability means.
The most simple example is the harmonic oscillation- $V=-{1 \over 2}kx^2 $.In Newtonian mechanics, for a point of stability,you have that $F(r_0)=0 $ where $r_0 $ is the point of stability. That is equal(if the $\nabla \times \bar F$ is zero) to the curve of the potential. So we have to find whether the $$\nabla V(r_0)=0 $$ gives us minimum values for the potential $V(r_0)$ or maximum values. Then, in a diagram V=V(r) the maximum obviously gives you, for a little transposition dr, the body moving away from an unstable point of stability. In the other hand, a minimum will give for the same transposition, a movement inside that like-a-well potential unless more energy is given to the body than the topical minimum has. Here's a picture:
It is useful and maybe fundamental to see this diagram in phase space(space of velocity and coordinate). Here:
The problem with this method is that if the scalar function $V(r)$ is a function of many variables(more than one) the mathematics require a lot of work.
So we use the method of disturbance(note: maybe my translation to English here is wrong). That means to take the force $F$ and write the Taylor expansion around a point(like mentioned in another answer) at distance e:
$$F(x_0+e)=F(x_0) + \left({dF \over dx}\right)|_{x=x_0}e + {1 \over 2}\left({d^2 F \over dx^2}|_{x=x_0}\right)e^2 +... =0$$. From here we take a linear differential equation whose solution is the behaviour of the particle
around $x_0$.
And next, if we have a Lagrangian $L=T-V $ where T the kinetic energy, with the general coordinates q in relation with the Cartesian are not a function of time, we have:
$$T=\sum_{i=1}^n \sum_{j=1}^n a_{ij}(q_1 ... q_n)\dot q_i \dot q_j $$ and $$V=V(q_1 ... q_n) $$ the equations of Lagrange are: $$\sum (q_1 .. q_n) \ddot q_i + \sum \sum b_{ijk}(q_1 ... q_n)\dot q_i \dot q_j + {\partial V \over \partial {q_k}}=0 $$ and k=0,1,2..n
For equilibrium solution the terms with derivatives over time are zero and so: $$\left.{ \partial V \over \partial {q_k}} \right|_{q_0} =0 $$.
For a little movement around the equilibrium point we can then prove:
$$L=\left(\sum_{i=1}^n m_{ij}\ddot χ_i + b_{ij}χ_i\right)=0 $$
Finally, who do we study the stability of circular orbits?
We have, for the equilibrium points: $$F(r)+{L^2 \over mr^3}=0 --> {dV(r) \over dr}=0 $$
For stable trajectory $V(r=r_0)''>0 $ where $V''$ is the second derivative over r of the potential. from here we can prove that:$$V''|_{r_0}=-F'|_{r_0} +{3 \over r_0}{L^2 \over mr{_0}^3}>0 -->{F'|_{r_0} \over F|_{r_0} + }+ {3 \over r_0 }>0$$. That's the condition for stable trajectories.
I know that's a lot, but I hope it helps.
What you just did was to find a condition for attractive power-law forces to have stable orbits where stable means they remain bounded when perturbed around the circular orbit. You got the correct result.
The Bertrand's Theorem though says something different: the only forces whose bounded orbits imply closed orbits are the Hooke's law and the attractive inverse square force. A closed orbit is one which the particle repeats their momentum and position after some finite time - it closes in the phase space.
The idea behind the proof of Bertrand's Theorem is to consider a perturbed orbit and then calculate the periods of the angular revolution and the radial oscillations. If these periods are commensurable then the orbit is closed. You can find the proof here.
Best Answer
The possible trajectories of a particle subject to an inverse-cube force $F = - k/r^3$ can actually be derived exactly; they are known as Cotes's spirals. Depending on the relative values of the particle's angular momentum $\ell$, its mass $m$, and the constant of proportionality $k$ in the force law, they take on the form $$ r(\theta) = \begin{cases} (A \cos C \theta + B \sin C \theta)^{-1} & km < \ell^2 \\ (A \cosh C \theta + B \sinh C \theta)^{-1} & km > \ell^2 \\ (A + B \theta)^{-1} & km = \ell^2 \end{cases} $$ where $$ C = \sqrt{\left| \frac{ m k}{\ell^2} - 1 \right|} $$ and $A$ and $B$ are determined by the initial conditions of the trajectory.
It is not hard to see that almost all of these functions will either have $r \to \infty$ for some finite value of $\theta$, or $r \to 0$ as $\theta \to\infty$, or both. The only case in which $r$ is bounded and does not reach $r = 0$ is when $km = \ell^2$ and $B = 0$, which corresponds to circular motion. Most perturbations from this trajectory will either involve changing $\ell$ (if the particle is given an extra tangential "push"), or changing $B$ (if the particle is given a purely radial push, since now we must have $dr/d\theta = 0$.) Thus, most perturbations will lead to the particle either spiraling in to $r = 0$ or flying out to $r \to \infty$.
Thinking about this in terms of the effective potential: to have a circular orbit in an inverse-cube field, you must have $\ell = \sqrt{km}$. A perturbation will either change $\ell$ or leave $\ell$ the same. If $\ell$ is changed from its initial value, then the effective potential becomes $U_\text{eff}(r) = Q/r^2$ for some value of $Q$, which has no maxima or minima; the radial motion must either go to $0$ or $\infty$. If $\ell$ is unchanged, then we must have $dr/dt = 0$, and the effective 1-D problem is that of a particle moving with some initial velocity in a potential $U_\text{eff} = 0$. Again, this radial motion must either go to $0$ or $\infty$.