I have more or less understood the underlying theory of the Lagrange multiplier method (by using the Implicit Function Theorem). Now, I try to extend this understanding to the general case, where we have more than one constraint. For example we try to maximize/minimize $f(x)$ subject to $g(x)=0$ and $h(x)=0$. As far as I can see, what we should do in this case is simply to build the Lagrange function
$$L(x,\alpha,\beta) = f(x) + \alpha g(x) + \beta h(x) $$ and then try to maximize/minimize this function, with respect to constraints $g(x)=0$ and $h(x)=0$.
To justify this form of the Lagrange function $L(x,\alpha,\beta)$, I thought the following: Assume that we have a point $x'$ which satisfies $g(x')=0$ and $h(x')=0$. If this point is an extreme point on both constraints $g(x)$ and $h(x)$, then it is $$\nabla f(x') = \lambda_1 \nabla g(x')$$ $$\nabla f(x') = \lambda_2 \nabla h(x')$$. We can unify these in a single equation as: $$\nabla f(x') -\dfrac{\lambda_1}{2}\nabla g(x') – \dfrac{\lambda_2}{2}\nabla h(x') = \nabla f(x') + \alpha \nabla g(x') + \beta \nabla h(x')= 0$$
This partially justifies $L(x,\alpha,\beta) = f(x) + \alpha g(x) + \beta h(x)$ for me: Calculate $\nabla_{x}L(x,\alpha,\beta)$, set it equal to zero and solve it; by using $g(x)=0$ and $h(x)=0$ as well.
But what disturbs me is that we could not able to find an analytic solution to this most of the time. I have prepared a rough sketch to show it:
Here, the constrained extreme points for both $g(x)=0$ and $h(x)=0$ are distinct ($x'$ and $x''$). The only points which satisfy both constraints at the same time are $A$ and $B$. And they are not the extreme points of the both constraint surfaces, it is $\nabla f(A) \neq \alpha \nabla g(A)$ for any $\alpha$ for example. So, my question is, what good is the Lagrangian function $L(x,\alpha,\beta) = f(x) + \alpha g(x) + \beta h(x)$ in such a case? It does not provide an analytic solution for such cases, then what is the point of the Lagrange function and the coefficients $\alpha$ and $\beta$ now? Does this form constitute a good structure for numerical optimization algorithms or what? I am confused about that.
Thanks in advance.
Best Answer
The sentence "If this point is an extreme point on both constraints $g(x)$ and $h(x)$, then it is $$\nabla f(x') = \lambda_1 \nabla g(x'),\qquad\nabla f(x') = \lambda_2 \nabla h(x')\ {\rm "}$$ is wrong. You only can say that $$\nabla f(x') = \lambda_1 \nabla g(x')+\lambda_2 \nabla h(x')\ .$$