The other answers here are in the spirit of what you can do, but allow me to elaborate a little more.
To understand if the trajectory of the movement under a potential $V $ is stable or not you have to understand what this stability means.
The most simple example is the harmonic oscillation- $V=-{1 \over 2}kx^2 $.In Newtonian mechanics, for a point of stability,you have that $F(r_0)=0 $ where $r_0 $ is the point of stability. That is equal(if the $\nabla \times \bar F$ is zero) to the curve of the potential. So we have to find whether the $$\nabla V(r_0)=0 $$ gives us minimum values for the potential $V(r_0)$ or maximum values. Then, in a diagram V=V(r) the maximum obviously gives you, for a little transposition dr, the body moving away from an unstable point of stability. In the other hand, a minimum will give for the same transposition, a movement inside that like-a-well potential unless more energy is given to the body than the topical minimum has. Here's a picture:
It is useful and maybe fundamental to see this diagram in phase space(space of velocity and coordinate). Here:
The problem with this method is that if the scalar function $V(r)$ is a function of many variables(more than one) the mathematics require a lot of work.
So we use the method of disturbance(note: maybe my translation to English here is wrong). That means to take the force $F$ and write the Taylor expansion around a point(like mentioned in another answer) at distance e:
$$F(x_0+e)=F(x_0) + \left({dF \over dx}\right)|_{x=x_0}e + {1 \over 2}\left({d^2 F \over dx^2}|_{x=x_0}\right)e^2 +... =0$$. From here we take a linear differential equation whose solution is the behaviour of the particle
around $x_0$.
And next, if we have a Lagrangian $L=T-V $ where T the kinetic energy, with the general coordinates q in relation with the Cartesian are not a function of time, we have:
$$T=\sum_{i=1}^n \sum_{j=1}^n a_{ij}(q_1 ... q_n)\dot q_i \dot q_j $$ and $$V=V(q_1 ... q_n) $$ the equations of Lagrange are: $$\sum (q_1 .. q_n) \ddot q_i + \sum \sum b_{ijk}(q_1 ... q_n)\dot q_i \dot q_j + {\partial V \over \partial {q_k}}=0 $$ and k=0,1,2..n
For equilibrium solution the terms with derivatives over time are zero and so: $$\left.{ \partial V \over \partial {q_k}} \right|_{q_0} =0 $$.
For a little movement around the equilibrium point we can then prove:
$$L=\left(\sum_{i=1}^n m_{ij}\ddot χ_i + b_{ij}χ_i\right)=0 $$
Finally, who do we study the stability of circular orbits?
We have, for the equilibrium points: $$F(r)+{L^2 \over mr^3}=0 --> {dV(r) \over dr}=0 $$
For stable trajectory $V(r=r_0)''>0 $ where $V''$ is the second derivative over r of the potential. from here we can prove that:$$V''|_{r_0}=-F'|_{r_0} +{3 \over r_0}{L^2 \over mr{_0}^3}>0 -->{F'|_{r_0} \over F|_{r_0} + }+ {3 \over r_0 }>0$$. That's the condition for stable trajectories.
I know that's a lot, but I hope it helps.
What you just did was to find a condition for attractive power-law forces to have stable orbits where stable means they remain bounded when perturbed around the circular orbit. You got the correct result.
The Bertrand's Theorem though says something different: the only forces whose bounded orbits imply closed orbits are the Hooke's law and the attractive inverse square force. A closed orbit is one which the particle repeats their momentum and position after some finite time - it closes in the phase space.
The idea behind the proof of Bertrand's Theorem is to consider a perturbed orbit and then calculate the periods of the angular revolution and the radial oscillations. If these periods are commensurable then the orbit is closed. You can find the proof here.
Best Answer
The effective potential in your case is
$$V_{\rm eff}(r)=\frac{L^2}{2mr^2}-V_{0}\exp(-\lambda^2r^2)$$
and if we differentiate
$$V^{\prime}_{\rm eff}(r)=-\frac{L^2}{mr^3}+2\lambda^2V_{0}r\exp(-\lambda^2r^2)$$
This vanishes if
$$L^2=2m\lambda^2V_{0}r^4\exp(-\lambda^2r^2)$$
as you've already observed. Now lets define $C\equiv2m\lambda^2V_{0}$ and $f(r)\equiv Cr^4\exp(-\lambda^2r^2)$, so our equation is
$$L^2=f(r)$$
This equation does not have a solution that can be expressed explicitly using elementary functions. For $L\leq L_{\rm max}$ (which you're asked to find in (b)), it has four solutions. Two positive solutions: one corresponds to a stable equilibrium and the other to an unstable one, and two mirrored negative solutions that are discarded.