Your absolute max and min will be achieved at critical points in the domain's interior, or on the boundary of the domain.
To find critical points (if any) in the interior of $D$, set $f_x(x,y)=y^2(4-2x-y)$ and $f_y(x,y)=xy(8-2x-3y)$ equal to $0$, and assume that $x>0,y>0,$ and $x+y<6$. Since $x,y\neq 0$ and $$y^2(4-2x-y)=0\\xy(8-2x-3y)=0,$$ it follows that $$4-2x-y=0\\8-2x-3y=0.$$ Solving this system for $y$ yields $y=2$, and back-substitution tells us that $x=1$. Indeed, $x+y=3<6$ and $x,y>0$, so this critical point is in the interior of $D$.
You've seen that $f(0,6)=f(6,0)=f(0,0)=0$, so it remains to check the boundary curves of $D$. On the lines $x=0$ and $y=0$, $f$ is constantly zero. All that's left is to check along the line $y=6-x$ for $0<x<6$. We do this by letting $$\begin{align}g(x) &:= f(x,6-x)\\ &= 4x(6-x)^2-x^2(6-x)^2-x(6-x)^3\\ &= (4x-x^2)(6-x)^2-(6x-x^2)(6-x)^2\\ &= -2x(6-x)^2.\end{align}$$ Then we check for critical points of $g$ by setting $g'(x)=0$ and looking for solutions with $0<x<6$. You should find such an $x$, and then you will also need to check $(x,6-x)$ as a potential location for the global max or min of $f$.
From your calculation, you have shown that there is no critical point in the interior of the half-disk, but the left endpoint of the semicircular boundary is one. The index of that point is
$$ D \ \ = \ \ f_{xx}·f_{yy} \ - \ (f_{xy})^2 \ \ = \ \ 2·(-2) \ - \ 0^2 \ = \ -4 \ \ < \ \ 0 \ \ , $$
which identifies $ \ (-1 \ , \ 0 ) \ $ as a saddle-point. This makes sense since the function is "concave upward" in the $ \ x-$ direction about $ \ x \ = \ -1 \ $ , due to the $ \ x^2 \ + \ 2x \ = \ (x + 1)^2 \ - \ 1 \ $ terms, but is "concave downward" in the $ \ y-$ direction because of the $ \ -y^2 \ $ term. As we will need it for comparisons later, we compute that $ f(-1,0) \ = \ (-1)^2 \ + \ 2(-1) \ - \ 0^2 \ = \ -1 \ \ . $
The lower boundary of the region is the $ \ x-$ axis over the interval $ \ [ \ -1 \ , \ 1 \ ] \ $ . Inserting $ \ y \ = \ 0 \ $ into the function expression gives $ \ f(x,0) \ = \ (x + 1)^2 \ - \ 1 \ \ . $ We have already found $ \ f(-1,0) \ $ to be the minimum on this line segment; the right endpoint has the value $ f(1,0) \ = \ (1+1)^2 \ - \ 1 \ = \ 3 \ \ . $
It remains to examine the upper semicircular boundary. There are a number of ways we might set this up, but to avoid having to work with a square-root or trigonometric identities, we will use the circle equation to write $ \ y^2 \ = \ 1 \ - \ x^2 \ $ and express our function on this section of the boundary as
$$ \ \phi(x) \ = \ (x + 1)^2 \ - \ 1 \ - \ (1 - x^2) \ \ = \ \ (x + 1)^2 \ + \ x^2 \ - \ 2 \ \ . $$
[This gives us the correct values for the function at the endpoints of the region:
$$ \phi(-1) \ \ = \ \ ([-1] + 1)^2 \ + \ (-1)^2 \ - \ 2 \ \ = \ -1 \ \ \ , \ \ \ \phi(1) \ \ = \ \ (1 + 1)^2 \ + \ 1^2 \ - \ 2 \ \ = \ \ 3 \ \ . \ ] $$
We find $ \ \phi'(x) \ = \ 2·(x + 1) \ + \ 2x \ \ = \ \ 0 \ \ \ \Rightarrow \ \ \ 4x \ \ = \ -2 \ \ \ \Rightarrow \ \ \ x \ = \ -\frac{1}{2} \ \ , $ for which $ \phi \left( -\frac{1}{2} \right) \ \ = \ \ \left( \ \left[-\frac{1}{2} \right] + 1 \ \right)^2 \ + \ \left(-\frac{1}{2} \right)^2 \ - \ 2 \ \ = \ \ \frac{1}{4} \ + \ \frac{1}{4} \ - \ 2 \ \ = \ -\frac{3}{2} \ \ . $
Hence, we observe that the absolute maximum of the function on this closed half-disc is $ f(1,0) \ = \ 3 \ $ and the absolute minimum is $ f \left( -\frac{1}{2} \ , \ \frac{\sqrt{3}}{2} \right) \ = \ -\frac{3}{2} \ \ . $ The saddle point at $ \ (-1 \ , \ 0 ) \ $ has no special role in this region.
Best Answer
$$f(x,y)=x^2+xy+y^2 = \dfrac{1}{2}((x+y)^2+x^2+y^2)\le \dfrac{1}{2}(x^2+y^2+2(x^2+y^2))\le \dfrac{3}{2}$$
The extrema occurs at $(x,y)=(\pm \frac{1}{\sqrt{2}},\pm \frac{1}{\sqrt{2}})$