First of all, I think that by ”determine the constraint”, it actually means determine the forces of constraint for the considered motion. Also, by ”Lagrange equation of the second kind”, it means that you should use the so-called Lagrange’s $\lambda$-method.
Let us consider the schematics below (which differ very little from your initial drawing). I believe it is much easier and perhaps elegant to work with some generalised coordinates, rather than with the usual cartesian ones. After all, one of the beauties of the Lagrangian method lies in this aptitude.
In my version, $r$ denotes the distance between the ground and the particle along the inclined plane/slope. Also, let us presume that for now, we do not know if the particle can or cannot sink into the surface of the wedge. Hence the presence of the $z$ coordinate. These two coordinates are the ones that really describe the particle motion. Then, if the $z$ coordinate can vary, then the particle height above the ground will be $r\sin\theta - z\cos\theta$. Taking this into account, its potential energy would be:
\begin{equation}
V = mg(r\sin\theta - z\cos\theta)
\end{equation}
But, in order to keep the particle on the surface, its $z$ coordinate must be zero. This leads to the holonomic constraint:
\begin{equation}
f(z) = z = 0.
\end{equation}
This constraint being holonomic, it enables a constraint potential $\lambda f(z)$ which can be included into our initial potential energy:
\begin{equation}
V(z) = mg(r\sin\theta - z\cos\theta) + \lambda z.
\end{equation}
The Kinetic Energy of the particle is:
\begin{equation}
T(\dot r, \dot z) = \frac{1}{2}m(\dot r^2 + \dot z^2),
\end{equation}
so the motion Lagrangian would in our case be:
\begin{equation}
L(r,\dot r,z, \dot z) = \frac{1}{2}m(\dot r^2 + \dot z^2) - mg(r\sin\theta - z\cos\theta) - \lambda z.
\end{equation}
This Lagrangian leads to two equations of motion,
\begin{equation}
\frac{d}{dt}\frac{\partial L}{\partial \dot r} - \frac{\partial L}{\partial r}= 0 \Leftrightarrow m\ddot r = mg\sin\theta
\end{equation}
\begin{equation}
\frac{d}{dt}\frac{\partial L}{\partial \dot r} - \frac{\partial L}{\partial r}= 0 \Leftrightarrow m\ddot z + mg\cos\theta = \lambda.
\end{equation}
But, our initial $f(z)$ constraint implies that $\ddot z = 0$. So we end up with the following two equations of motion :
\begin{equation}
\ddot r = g\sin\theta
\end{equation}
\begin{equation}
\lambda = mg\cos\theta
\end{equation}
where in fact, the second equation of motion corresponds to the force of constraint, where $\mathbf{\lambda}$ is the normal force exerted by the wedge surface on the particle; the first equation corresponds to the sliding motion of the particle.
In the end, the lagrange multiplier $\lambda$ is the constraining force of the sliding motion: the one needed to keep the particle on the surface.
Best Answer
Your sign is wrong. You should get $m\ddot x+\nabla V(x)=0$ from the Euler-Lagrange equation $0=-\frac{d}{dt}\frac{∂L}{∂\dot x}+\frac{∂L}{∂x}$.
As $H=T+V$ is a first integral, you get $$ \frac{m}{k}\dot x^2+x^2+\fracλ{2k}x^4=R^2=const. $$ which motivates to set $$\dot x=R\omega\cos(ωu),~~~ y=g(x)=x\sqrt{1+\fracλ{2k}x^2}=R\sin(ωu),$$ $ω^2=\frac{k}m$. Now $g$ is smooth and monotonically increasing, so the inverse function $f=g^{-1}$ exists and is differentiable. With $x=f(R\sin(ωu))$ leading to a second form for $$\dot x=f'(R\sin(ωu))ωR\cos(ωu)\dot u$$ we get $$ \dot u=\frac1{f'(R\sin(ωu))} $$ which reduces the problem to a simple scalar ODE for the angular factor $u$ with a positive right side.