To say that the test is inconclusive when the determinant $f_{xx} f_{yy} - f_{xy}^2$ is zero at a point is to say just that: The test doesn't tell us anything, so if we want to determine the type of the critical point, we must do a little more. (Indeed, the functions $(x, y) \mapsto x^4 + y^4$, $(x, y) \mapsto x^4 - y^4$, and $(x, y) \mapsto -x^4 - y^4$ all have a critical point at $(0, 0)$ with zero Hessian determinant, but the three functions respectively have a minimum, a saddle point, and a maximum there.)
First, note that something like this issue already occurs for single-variable functions. Checking shows that $g(x) := x^4$ has a critical point at $x = 0$, and computing the second derivative there gives $g''(0) = 0$, so we cannot conclude whether $g$ has a minimum, a maximum, or neither, at that point. We can still determine the type of critical point, however, by observing that $g(x) > 0$ for any $x \neq 0$, and so the critical point must be an (isolated) minimum.
In the case of our two-variable function $f(x, y)$, we can proceed as follows: Computing gives that $$f(x, 0) = x^4, $$ which we've already said has an isolated minimum at $x = 0$. On the other hand, $$f(0, y) = y^4 - y^2 $$ Applying the (single-variable) Second Derivative Test gives $\frac{d^2}{dy^2} [f(0, y)] = -2$, so $f(0, y)$ has an isolated maximum at $y = 0$. So, $f(x, y)$ takes on both positive and negative values in any open set containing $(0, 0)$, so it must be a saddle point.
Best Answer
I think you should re-look at your hessian, the hessian is often a symmetric matrix, and indeed it is in this case too.
Beyond that, if the hessian is degenerate or 0, you should think along the same lines as having the first and second derivatives of a one-variable function being zero, and what information this could give you (basically none).
So, what is to be done? Looking at your hint with lines, it's obvious that the function isn't going to get smaller than the value at 0 anywhere (or it would have on one of the lines), but it could be 0 along a curve. I think the only thing left to you is to be clever. If, for example, we look at what happens by taking $y=x^2$, we find that $$F(x,x^2) = 3x^4-4x^4+x^4=0$$ This is true for all x, including zero, so the point at zero is not a local min.