multivariable-calculus
Let $f:{\Bbb R^2}\to{\Bbb R}$ be a function such that
$$
f(x,y)=5x^2+xy^3-3x^2y.
$$
Is $(0,0)$ a local maximum, local minimum or a saddle point?
Calculation shows that $(0,0)$ is a degenerate critical point. One can of course use MATLAB or Mathematic to have a graph to see the property of this point.
Your intuitive picture is probably as follows: Imaging filling water into the hollow at the origin. As the water rises, since there are points lower than the origin, you expect the water to start spilling over into the lower lying areas at some point. And that point would need to be a critical point! The way to resolve the paradox is to note that before any spilling over occurs, the lake will in fact extend off to infinity. Look at a cross section of your function graph for fixed $y$, as $y$ starts at $0$ and then grows. In other words, study the function $x\mapsto x^2+y^2(1-x)^3$ as $y^2$ grows. You will see a local minimum, starting at the origin and approaching $x=1$ from below, and a local maximum, starting at infinity and approaching $x=1$ from above. As $y^2$ grows, the value at the local minimum grows, while the value at the local maximum decreases. So the aforementioned lake will extend out to $(1,\pm\infty)$ and then start to spill over the edge (the local maximum) at infinity before it spills over anywhere else.
A picture is tremendously helpful. If you have access to maple, try this:
plots[animate](plot,
[x^2+y^2*(1-x)^3, x = 0 .. 2, view = [0 .. 2, -1 .. 2]], y = 0 .. 10);
Or you can take a look at this static plot of $f(x,y)$. It is easy to see how $(0,0)$ is a local, but not a global, maximum:
To say that the test is inconclusive when the determinant $f_{xx} f_{yy} - f_{xy}^2$ is zero at a point is to say just that: The test doesn't tell us anything, so if we want to determine the type of the critical point, we must do a little more. (Indeed, the functions $(x, y) \mapsto x^4 + y^4$, $(x, y) \mapsto x^4 - y^4$, and $(x, y) \mapsto -x^4 - y^4$ all have a critical point at $(0, 0)$ with zero Hessian determinant, but the three functions respectively have a minimum, a saddle point, and a maximum there.)
First, note that something like this issue already occurs for single-variable functions. Checking shows that $g(x) := x^4$ has a critical point at $x = 0$, and computing the second derivative there gives $g''(0) = 0$, so we cannot conclude whether $g$ has a minimum, a maximum, or neither, at that point. We can still determine the type of critical point, however, by observing that $g(x) > 0$ for any $x \neq 0$, and so the critical point must be an (isolated) minimum.
In the case of our two-variable function $f(x, y)$, we can proceed as follows: Computing gives that $$f(x, 0) = x^4, $$ which we've already said has an isolated minimum at $x = 0$. On the other hand, $$f(0, y) = y^4 - y^2 $$ Applying the (single-variable) Second Derivative Test gives $\frac{d^2}{dy^2} [f(0, y)] = -2$, so $f(0, y)$ has an isolated maximum at $y = 0$. So, $f(x, y)$ takes on both positive and negative values in any open set containing $(0, 0)$, so it must be a saddle point.
Best Answer
Informally, you only need to look at the function when $x$ and $y$ are small. In that case, the 3rd order element $x^2y$ and the 4th order elements $xy^3$ can be neglected in front of the 2nd order element $5x^2$. So $f(x,y) \approx 5x^2$, which looks like a parabolic cylinder in the $Oy$ direction.
More formally, replace $f$ by the first terms of a Taylor series : $$ f(x,y) = f(0,0) + \frac {\partial f} {\partial x} x + \frac {\partial f} {\partial y} y + \frac {\partial^2 f} {\partial x^2} x^2 + \frac {\partial^2 f} {\partial x \partial y} x y + \frac {\partial^2 f} {\partial y^2} y^2 +....$$
keeping only the terms of the smallest degree, which gives you an approximation of the function in the neighborhood of $(0,0)$.
In your case, your $f$ is a polynomial, so the Taylor expansion is itself.