I'm learning the math behind SVMs (Support Vector Machines) from here. I understood the intuition behind Lagrange multipliers. But what I don't understand is that, why Lagrange multiplier is considered as a variable in Lagrangian?
Isn't it just a constant by which the gradient vectors of the objective function (for which the optimization is to be done) and the subject function (constraint function) vary in length?
Here is the equation of the Lagrangian:
$$L(x, y, \lambda) = f(x,y) – \lambda g(x, y)$$
where $f(x, y)$ is objective function and $g(x, y)$ is constraint function. Now applying gradient both sides:
$$\nabla L(x, y, \lambda) = \nabla f(x,y) – \lambda \nabla g(x, y)$$
Here while applying $\nabla$, we've considered $\lambda$ as constant, even if we are considering Lagrangian ($L$) as a function of $x$, $y$ and $\lambda$.
What I'm missing here? Any intuitive explanation will help. Thanks!
EDIT
In Wikipedia page of this topic, $\lambda$ is considered as constant. Here is the direct quote from Wikipedia page.
The constant λ is required because although the two gradient vectors are parallel, the magnitudes of the gradient vectors are generally not equal. This constant is called the Lagrange multiplier. (In some conventions λ is preceded by a minus sign).
Best Answer
The minimal value point $(x_0,y_0)$ must satisfies two conditions: $$g(x_0,y_0)=0\qquad \mbox{and}\qquad \nabla g(x_0,y_0)=\lambda_0 \nabla f(x_0,y_0)$$ where $\lambda_0$ is constant.
Define $L(x,y)=f(x,y)-\lambda_0 g(x,y)$, then $\nabla g(x_0,y_0)=\lambda_0 \nabla f(x_0,y_0)$ is equivalent to $$L_x(x_0,y_0)=L_y(x_0,y_0)=0$$ and in form:$$L_{\lambda_0}(x_0,y_0)=g(x_0,y_0)=0$$ so consider $\lambda$ as variable is only in order to guarantee the constraint condition $g(x_0,y_0)=0$.