I will write about the function
$$f(x,y)=x^3+y^3-3xy$$
for $x,y\in[0,4]$.
This is basically the same problem, it seems a bit more symmetric after the change of variables.
Local extrema:
You correctly found
\begin{align*}
f_x &= 3x^2-3y\\
f_y &=-3x+3y^2
\end{align*}
and identified critical points as $(0,0)$ and $(1,1)$.
We can calculate also the second derivatives: $f_{xx}=6x$, $f_{xy}=-3$, $f_{xy}=6y$. We have
\begin{align*}
D_1&=6x\\
D_2&=\begin{vmatrix}6x&-3\\-3&6y\end{vmatrix}=36xy-9
\end{align*}
We see that for $x=y=1$ we get $D_1>0$ and $D_2>0$. So the corresponding matrix is
positive definite
and this is a
local minimum.
For $x=y=0$ we have $D_1=0$, so we cannot say from this whether it is maximum or minimum. But it is relatively easy to see that it is neither of them, since we can find close to $(0,0)$ points with positive values and points with negative values. (It suffices to check $f(\varepsilon,0)=\varepsilon^3$ for small $\varepsilon$.)
We can also ask Wolfram Alpha to check for local extrema of x^3+y^3-3xy to verify whether our calculations are correct.
Extrema on $K$
But we are interested in the extrema on the set $K$, not in the local extrema of the function $f$.
Since this set is closed and bounded and the function $f$ is continuous, it is guaranteed that the function $f$ attains maximum and minimum on $K$.
If the extrema are in the interior, then they are local extrema. So one candidate is $(1,1)$. We have $f(1,1)=-1$.
Now we check what happens on the boundary. We have $f(0,0)=0$, $f(4,0)=f(0,4)=4^3=64$ and $f(4,4)=4^3+4^3-3\cdot4^2=4^2(4+4-3)=80$.
What happens on the sides of the square?
- For $y=0$ we have $f(x,0)=x^3$. This function is clearly increasing for $x\in[0,4]$. So on this side we have minimum for $x=0$ and maximum for $x=4$.
- For $y=4$ we have $f(x,4)=x^3-12x+64$. We can check that this function is decreasing for $x\in[0,2]$ and increasing for $x\in[2,4]$. So the values interesting for us are $f(0,4)=64$, $f(2,4)=48$ and $f(4,4)=80$.
- The other two sides behave symetrically.
So we see that the values attained on the boundary are between $f(0,0)=0$ and $f(4,4)=80$.
The conclusion is that the maximum on $K$ is $f(4,4)=80$ and the minimum is $f(1,1)=-1$.
Again we can check WA: extrema x^3+y^3-3xy for 0<=x<=4, 0<=y<=4.
I suggest to take geo method directly which is easy to understand. check above picture, you need to find the circle(center is fixed) max and min R in a triangle region that is very straightforward.
edit 1:(add more details)
$f(x,y)=(x-2)^2+y^2+5 \implies (x-2)^2+y^2=f(x,y)-5=R^2$
you want to find max and min of $f(x,y)$ is same to find max and min of $R$.
note the circle have to be inside the triangle so max of $R$ is $2$ ,min of R is $0$ which give $f_{max}=4+5=9, f_{min}=0+5=5$
if op insist on pure Lagrange multiplier method, it has much more things to write and you have three boundary equations . Anyway, if you don't like geo method, simply ignore it. it is true that we can't use this simple method if $f(x,y)$ is not the circle.
Best Answer
Your absolute max and min will be achieved at critical points in the domain's interior, or on the boundary of the domain.
To find critical points (if any) in the interior of $D$, set $f_x(x,y)=y^2(4-2x-y)$ and $f_y(x,y)=xy(8-2x-3y)$ equal to $0$, and assume that $x>0,y>0,$ and $x+y<6$. Since $x,y\neq 0$ and $$y^2(4-2x-y)=0\\xy(8-2x-3y)=0,$$ it follows that $$4-2x-y=0\\8-2x-3y=0.$$ Solving this system for $y$ yields $y=2$, and back-substitution tells us that $x=1$. Indeed, $x+y=3<6$ and $x,y>0$, so this critical point is in the interior of $D$.
You've seen that $f(0,6)=f(6,0)=f(0,0)=0$, so it remains to check the boundary curves of $D$. On the lines $x=0$ and $y=0$, $f$ is constantly zero. All that's left is to check along the line $y=6-x$ for $0<x<6$. We do this by letting $$\begin{align}g(x) &:= f(x,6-x)\\ &= 4x(6-x)^2-x^2(6-x)^2-x(6-x)^3\\ &= (4x-x^2)(6-x)^2-(6x-x^2)(6-x)^2\\ &= -2x(6-x)^2.\end{align}$$ Then we check for critical points of $g$ by setting $g'(x)=0$ and looking for solutions with $0<x<6$. You should find such an $x$, and then you will also need to check $(x,6-x)$ as a potential location for the global max or min of $f$.