Lagrange Multipliers – probability distribution with “Between 0 and 1” restrictions

lagrange multipliernumerical methodsoptimization

I am attempting to find the maximum and minimum of the second moment for a discrete probability distribution with a certain mean. In other words, find the maximum and minimum of

$$f = 0^2p_0 + 1^2p_1 + 2^2p_2 + 3^2p_3 + 4^2p_4$$

Given restraints:

$$p_0 + p_1 + p_2 + p_3 + p_4 = 1$$

and

$$0p_0 + 1p_1 + 2p_2 + 3p_3 + 4p_4 = A$$

Where A is some fixed mean. I am stuck on the fact that each $p_n$ must be between 0 and 1. How can I express that in terms of Lagrange Multipliers? Does it involve slack variables? Thank you.

Best Answer

Let $f(p)=\sum_{k=0}^4 k^2 p_k$.

Let $g_k(p)=-p_k$ and $\overline{g}_k(p)=p_k-1,k=0,1,\dots,4$. (Bars in this answer aren't conjugates, it's just notation.)

Let $h_0(p)=\sum_{k=0}^4 p_k-1,h_1(p)=\sum_{k=0}^4 k p_k - A$.

In this notation, the KKT stationarity condition is

$$\pm \nabla f(p)=\sum_{k=0}^4 \left ( \mu_k \nabla g_k(p) + \overline{\mu}_k \nabla \overline{g}_k(p) \right ) + \sum_{k=0}^1 \lambda_k \nabla h_k(p)$$

with the sign depending on whether you are maximizing (+) or minimizing (-).

You also have the constraints of course (called "primal feasibility" conditions), which in this notation read:

$$h_k=0,k=0,1 \\ g_k \leq 0,k=0,1,\dots,4 \\ \overline{g}_k \leq 0,k=0,1,\dots,4.$$

Finally you have the other two KKT conditions. First there's dual feasibility:

$$\mu_k \geq 0,k=0,1,\dots,4\\ \overline{\mu}_k \geq 0,k=0,1,\dots,4.$$

Second there's complementary slackness:

$$\mu_k g_k=0,k=0,1,\dots,4 \\ \overline{\mu}_k \overline{g}_k=0,k=0,1,\dots,4.$$

In total you have 17 equality constraints in 17 unknowns, so the equation counting checks out.

Related Question