I learned Lagrange multipliers as an undergrad and applied them with reckless abandon. Any constrained optimization problem could be solved by Lagrange multipliers. Now that I am older and wiser, I have found cases where Lagrange multipliers fail. It actually seems very rare that Lagrange multipliers should work, even.
Here is my understanding. If you want to compute $\inf_x f(x)$ subject to $g(x)=0$ then you define the Lagrangian function
$$L(x,\lambda)=f(x)+\lambda g(x)$$
Then $$\sup_\lambda L(x,\lambda)=\begin{cases}f(x) &\text{ if } g(x)=0\\ \infty&\text{ if }g(x)\neq 0\end{cases}$$
So $\inf_x\sup_\lambda L(x,\lambda)$ is literally $\inf_x f(x)$ given that $g(x)=0$.
The hope for Lagrange multipliers is that $\sup_\lambda \inf_x L(x,\lambda)=\inf_x\sup_\lambda L(x,\lambda)$ (that is, strong duality). We always have that $\sup_\lambda \inf_x L(x,\lambda)\leq \inf_x\sup_\lambda L(x,\lambda)$ (weak duality).
We can construct basic examples for when Lagrange multipliers don't work (i.e. strong duality fails). In fact, it seems very unlikely and special that strong duality holds. But in undergrad calculus classes, the issue of strong duality is never a problem. Why?
Best Answer
In undergrad you never use strong duality, you merely use that KKT conditions are often necessary conditions. Therefore, by using the Weierstrass extreme value theorem and solving the KKT conditions, you end up with a few candidate minima, and by comparing objective functions you end up with the optimal solution.
Looking back at my calculus book, the exercises all involved a single constraint, and the theorem had a condition that the gradient of that constraint had to be nonzero at the optimum. That means that the LICQ (see the link) was satisfied.