Your absolute max and min will be achieved at critical points in the domain's interior, or on the boundary of the domain.
To find critical points (if any) in the interior of $D$, set $f_x(x,y)=y^2(4-2x-y)$ and $f_y(x,y)=xy(8-2x-3y)$ equal to $0$, and assume that $x>0,y>0,$ and $x+y<6$. Since $x,y\neq 0$ and $$y^2(4-2x-y)=0\\xy(8-2x-3y)=0,$$ it follows that $$4-2x-y=0\\8-2x-3y=0.$$ Solving this system for $y$ yields $y=2$, and back-substitution tells us that $x=1$. Indeed, $x+y=3<6$ and $x,y>0$, so this critical point is in the interior of $D$.
You've seen that $f(0,6)=f(6,0)=f(0,0)=0$, so it remains to check the boundary curves of $D$. On the lines $x=0$ and $y=0$, $f$ is constantly zero. All that's left is to check along the line $y=6-x$ for $0<x<6$. We do this by letting $$\begin{align}g(x) &:= f(x,6-x)\\ &= 4x(6-x)^2-x^2(6-x)^2-x(6-x)^3\\ &= (4x-x^2)(6-x)^2-(6x-x^2)(6-x)^2\\ &= -2x(6-x)^2.\end{align}$$ Then we check for critical points of $g$ by setting $g'(x)=0$ and looking for solutions with $0<x<6$. You should find such an $x$, and then you will also need to check $(x,6-x)$ as a potential location for the global max or min of $f$.
You can reduce $f_x, f_y, f_z$ by dividing out the $3$ to:
$$x^2-3y-3z+9 = 0 \\ y^2-3x = 0 \\z^2-3x =0$$
Update
From $z = \pm y$, we substitute in the first equation and have:
$$x^2 - 3 y+ 3 (\pm~ y) + 9 = 0$$
This gives two cases to check:
$$x^2 - 6y + 9 = 0, x^2 + 9 = 0$$
The second does not work, for the first, we know that $x = \dfrac{1}{3} y^2$. Substitute and you get $y = 3, 1.63107$ and two imaginary roots that we can immediately ignore (real numbers only). Now use those two $y$ values, substitute to find the corresponding $x$ and $z$ values.
One of those has a solution $x = 0.886793$ and another solution $x = 3$. Find the corresponding $z$ values. All other choices/approaches lead to these roots and other imaginary roots that we can ignore.
So, we have the following roots:
$$(x, y, z) = (3,3,3) ~~\mbox{or} ~~ (0.886793, 1.63107, 1.63107)$$
Best Answer
Note that the denominator $x^2 + y^2 + 1 \ge 1$ for all $(x,y) \in \mathbb R^2$. So $f$ is well-defined and continuous. Next, we note that $f \ge -4$ for all $x,y$, since $$xy(xy+4) = (xy+2)^2 - 4.$$ So we know that a minimum must exist.* As for a maximum, we have for $|x| > 1$ $$f(x,x) > \frac{x^4}{2x^2 + 1} \ge \frac{x^4}{2(x^2+x^2)} = \frac{x^2}{4},$$ so $f$ has no global maximum.
To find the minimum, it is worth considering $$f(r \cos \theta, r \sin \theta) = \frac{r^2 \sin 2\theta (r^2 \sin 2\theta + 8)}{4 (r^2 + 1)}. \tag{1}$$ Regarding $r$ as fixed, we note $$\frac{\partial f}{\partial \theta} = 4 r^2 \cos 2\theta (4 + r^2 \sin 2\theta),$$ so any minimum would satisfy $\partial f/\partial \theta = 0$, implying for $\theta \in (-\pi, \pi]$ $$\theta \in \left\{ \pm \frac{\pi}{4}, \pm \frac{3\pi}{4}, \frac{1}{2} \arcsin \left(-\tfrac{4}{r^2} \right) \right \}.$$ Of these, clearly if $0 \le r < 2$, the last root is invalid. Otherwise, we can substitute back into Equation $(1)$ and show that there are no critical points for $r > 0$. The other possibilities are easily checked for critical points with respect to $r$, which I leave as an exercise.
*Note. Technically, it doesn't have to have a minimum, since a function like $e^x$ does not have a minimum although it is bounded below by $0$. But it does suggest searching for one.