You have $$\nabla f(x,y) = \begin{bmatrix} -x^2 + 1 \\ - 2y \end{bmatrix}.$$
So the critical points are $(-1,0)$ and $(1,0)$.
Now, the Hessian is
$$\nabla^2 f(x,y) = \begin{bmatrix} -2x & 0 \\ 0 & -2 \end{bmatrix}.$$
The eigenvalues of $\nabla^2 f(-1,0)$ are $2$ and $-2$. Thus, $\nabla^2 f(-1,0)$ is indefinite and $(-1,0)$ is a saddle point. The eigenvalues of $\nabla^2 f(1,0)$ are $-2$ and $-2$. Thus, $\nabla^2 f(1,0)$ is negative definite and $(1,0)$ is a maximum.
To find the critical points, solve the simultaneous equations
$$f_x=3x^2+9y \tag{1}$$
$$f_y=-3y^2+9x \tag{2}$$
From (2), without introducing additional solutions or losing any, we get $x=\tfrac{1}{3}y^2$, so by substituting into (1):
$$\tfrac{1}{3}y^4+9y=0 \iff \tfrac{1}{3}y(y+3)(y^2-3y+9)=0$$
The quadratic term has no real roots, so the only solutions are $y=0,-3$, so solutions are:
$$(x,y)\in\{(0,0),(3,-3)\}$$
To find the nature of the critical points, you will need the partial derivatives of $f(x,y)$. These are:
$$\begin{align}
f_{xx} &= \frac{\partial}{\partial x}f_x&=6x \\[2ex]
f_{yy} &= \frac{\partial}{\partial y}f_y&=-6y \\[2ex]
f_{xy} &= \frac{\partial}{\partial x}f_y&=9
\end{align}$$
Then conduct the second partial derivative test - as uniquesolution has commented, this involves the Hessian. In short, the sign of the determinant $D(x,y)$ and the sign of either $f_{xx}$ or $f_{yy}$ (your choice) determines the nature of the critical point. Wikipedia is a good source for this.
Now
$$D(x,y)=f_{xx}f_{yy}-(f_{xy})^2=-36xy-81$$
So for the two critical points:
- $(0,0)$: $D(0,0)=-81$ which is negative, so $(0,0)$ is a saddle point.
- $(3,-3)$: $D(3,-3)=(-36)(-9)-81=243$ which is positive, so look at the sign of $f_{xx}$ at $(3,-3)$ which is $6(3)=18$ and positive, so this point is a local maximum.
Addendum: Other Cases
A. $D(x,y)=0$
Many of the ideas in this section (and the first two examples) are taken from
https://www3.nd.edu/~eburkard/Teaching/20550%20F13/Second%20Derivatives%20Test.pdf.
When the determinant is zero, the Hessian matrix is said to be degenerate. I know of no higher-order (third derivatives or higher) test that can determine the nature of the critical points. Several examples will show that behaviour can vary markedly in cases where $f_x=f_y=D(x,y)=0$ at a critical point:
- $f(x,y)=-(x-y)^2$. Here there is a line of critical points $y=x$ along which $f(x,y)=0$. This is the maximum possible value of $f$.
- $f(x,y)=x^3+y^3$. In every nbhd of $(0,0)$ there are points where $f>0$ (wherever $(x,y)\text{ s.t. }x+y>0$) and points where $f<0$ (wherever $(x,y)\text{ s.t. }x+y<0$) so this is not a local min or max. It's not really a saddle point either.
- $f(x,y)=x^4+y^4$. It is clear that $(0,0)$ is a local minimum in this case.
B. $D(x,y)>0\text{ and }f_{xx}=0$
When $f_{xx}=0$ we have $D(x,y)=f_{xx}f_{yy}-(f_{xy})^2=0-(f_{xy})^2\le 0$, so this case is not possible.
Best Answer
$\nabla f=(10x+2y,2x+10y)=0\Rightarrow 5x+y=0 $ and $5y+x=0$
Therefore, correctly, the only critical point is (0,0).
Can you now classify it?