Alright I've figured it out.
The stationary functions of the augmented functional must satisfy not only the Euler-Lagrange equation, but also the additional natural boundary condition $H[y(\pm a)] = 0$, where $H[y] = -f + y'f_{y'}$ (this can be determined just by taking the Gateaux derivative of $\tilde{J}(y; a)$ and setting it equal to zero for all admissible directions).
In this case, the $H[y]$ is also just our Euler-Lagrange equation, and so $H[y(\pm a)] = 0$ tells us that $C$ equals zero, in which case our equation is reduced simply to
$$
y = 2\lambda\cos\theta
$$
after the parametrization $y' = \tan\theta$.
Method for solving a two-variable maximization problem with one equality constraint
Let $f$ and $g$ be continuously differentiable functions and let $c$ be a number. If the problem
$$\text{$\max_{x,y}f(x,y)$ subject to $g(x,y)=c$ and $(x,y)\in S$}$$
has a solution, then it may be found as follows:
- Find all the values of $(x,y,\lambda)$ for which
$$\begin{align*}&\text{(a) $(x,y)$ is an interior point of $S$}\\ &\text{(b) $(x,y,\lambda)$ satisfies the first-order conditions and the constraint:}\end{align*}$$
$$\begin{align*}L_x(x,y)&=0\\ L_y(x,y)&=0 \\ g(x,y)&=c. \end{align*}$$
- Find all the points $(x,y)$ in the interior of $S$ that satisfy
$$\begin{align*}g_x(x,y)&=0\\ g_y(x,y)&=0 \\ g(x,y)&=c. \end{align*}$$
If the set $S$ has any boundary points, find all the boundary points that solve the problem $\max_{x,y}f(x,y)$ subject to the two conditions $g(x,y) = c$ and $(x,y)$ is a boundary point of $S$.
The points $(x,y)$ you have found at which the value $f(x,y)$ is largest are the maximizers of the function $f$.
You have already carried out step 1. In many cases that will give your solution. However, you have to check points that do not satisfy the constraint qualification. A solution $(x^*,y^*)$ satisfies the constraint qualification if $g_x(x^*,y^*)\neq 0$ or $g_y(x^*,y^*)\neq 0$. Step 2 finds all the points in the constraint set that do not satisfy the constraint qualification, as they could potentially be solutions that are not found in step 1. Step 3 addresses your concern related to your previous question by also checking the boundary points of $S$.
In your case, steps 2 and 3 are as follows:
$g_x(x,y)=m\neq 0$ and $g_y(x,y)=n\neq 0$, so there are no points satisfying the conditions in step 2. (The constraint qualification is satisfied at all points.)
$S=\{(x,y)\ |\ x\geq 0, y\geq 0\}$, and the boundary points are where $x=0$ or $y=0$. When $x=0$ or $y=0$ we have $f(x,y)=0$. Since the point you found in step 1 gives $f(x,y)>0$, there is no need to go any further.
To address you first concern, once we have proved a maximum exist, then we know it must be found in step 1, 2, or 3. If a minimum exists, then we know it must satisfy the conditions in step 1, 2, or 3 with $\max$ replaced by $\min$. It is step 4 that then allows you to distinguish between which candidates are minimizers or maximizers.
To prove existence of a maximum and minimum usually involves application of the extreme value theorem. In your case the constraint set is compact (it is the line $g(x,y)=p$ for $x\geq 0$ and $y\geq 0$) and the objective function is continuous, so you know both a maximum and minimum exist.
In your case, the minimum is $0$, and this occurs at boundary points of $S$ satisfying the constraint, i.e. at the points $(0,p/n)$ and $(p/m,0)$.
Best Answer
Too long for a comment.
You missed a term in $J^{*}$; it should be $J^{*}=\int_0^{\pi/2}F^{*}(x,y_1,y_2,y_1',y_2',\lambda)\,dx$, where $$ F^{*}(x,y_1,y_2,y_1',y_2',\lambda)=2y_1y_2+(y_1')^2+(y_2')^2+\lambda(y_1'+y_2'-4x). \tag{1} $$ Now, write the Euler-Lagrange equations for $F^{*}$: \begin{align} &\frac{\partial F^{*}}{\partial y_1}-\frac{d}{dx}\left(\frac{\partial F^{*}}{\partial y_1'}\right)=0\implies 2y_2-2y_1''-\lambda'=0, \tag{2a} \\ &\frac{\partial F^{*}}{\partial y_2}-\frac{d}{dx}\left(\frac{\partial F^{*}}{\partial y_2'}\right)=0\implies 2y_1-2y_2''-\lambda'=0, \tag{2b} \\ &\frac{\partial F^{*}}{\partial \lambda}=0\implies y_1'+y_2'-4x = 0. \tag{2c} \end{align} Can you continue from here?