In order to get a feeling of what is possible you should look at polynomial functions $(x,y)\mapsto f(x,y)$ with $f(0,0)=f_x(0,0)=f_y(0,0)=0$. Such functions are their own Taylor expansions to start with, and the character of the citical point $(0,0)$ is immediately visible in the sign of $f(x,y)$ for various $(x,y)$ near $(0,0)$.
The generic case is that $f(x,y)=a x^2 + 2b x y + c y^2 +{\rm higher\ terms}$ with $ac -b^2\ne 0$. In this case you have a local max, a local min or a saddle point at $(0,0)$.
Consider now as an example the function
$$f(x,y)=(y-x^2)(y+x^2)\ .$$
Its zero set consists of the two parabolas $y=\pm x^2$, and it is easy to see that $f$ takes positive as well as negative values in the immediate neighborhood of $(0,0)$; whence we don't have a local extremum there. If $f$ were given in a more disguised form (and/or would have additional higher terms) you would have to look at the fourth degree Taylor polynomial of $f$ to find this out through analyzing partial derivatives.
This example tells you that in the case of a degenerate critical point heavy algebraic machinery could be needed for a final diagnosis of this point.
To say that the test is inconclusive when the determinant $f_{xx} f_{yy} - f_{xy}^2$ is zero at a point is to say just that: The test doesn't tell us anything, so if we want to determine the type of the critical point, we must do a little more. (Indeed, the functions $(x, y) \mapsto x^4 + y^4$, $(x, y) \mapsto x^4 - y^4$, and $(x, y) \mapsto -x^4 - y^4$ all have a critical point at $(0, 0)$ with zero Hessian determinant, but the three functions respectively have a minimum, a saddle point, and a maximum there.)
First, note that something like this issue already occurs for single-variable functions. Checking shows that $g(x) := x^4$ has a critical point at $x = 0$, and computing the second derivative there gives $g''(0) = 0$, so we cannot conclude whether $g$ has a minimum, a maximum, or neither, at that point. We can still determine the type of critical point, however, by observing that $g(x) > 0$ for any $x \neq 0$, and so the critical point must be an (isolated) minimum.
In the case of our two-variable function $f(x, y)$, we can proceed as follows: Computing gives that $$f(x, 0) = x^4, $$ which we've already said has an isolated minimum at $x = 0$. On the other hand, $$f(0, y) = y^4 - y^2 $$ Applying the (single-variable) Second Derivative Test gives $\frac{d^2}{dy^2} [f(0, y)] = -2$, so $f(0, y)$ has an isolated maximum at $y = 0$. So, $f(x, y)$ takes on both positive and negative values in any open set containing $(0, 0)$, so it must be a saddle point.
Best Answer
You found correctly the critical points: $$ \begin{cases} 3x^2+2yz=0 \\[4px] 3y^2+2xz=0 \\[4px] 2z+2xy=0 \end{cases} $$ gives $z=-xy$ and so the first and second equations become $$ \begin{cases} 3x^2-2xy^2=0 \\[4px] 3y^2-2x^2y=0 \end{cases} $$ If $x=0$, also $y=0$ and $z=0$. If $y=0$, then also $x=0$ and $z=0$. Assuming $x\ne0$ and $y\ne0$, we get $x=3/2$, $y=3/2$ and $z=-9/4$.
Indeed the Hessian matrix at $(0,0,0)$ has zero determinant, so it cannot be used for determining the character of this stationary point. However, you can look at what happens on particular restrictions: consider the line $$ \begin{cases} x=at\\[4px] y=bt\\[4px] z=ct \end{cases} $$ that leads us to study $$ g(t)=f(at,bt,ct)=(a^3+b^3+2abc)t^3+c^2t^2 $$ If $a=1$, $b=0$, $c=0$, we have $g(t)=t^3$.
Thus the origin is neither a point of maximum nor of minimum.