You have $$\nabla f(x,y) = \begin{bmatrix} -x^2 + 1 \\ - 2y \end{bmatrix}.$$
So the critical points are $(-1,0)$ and $(1,0)$.
Now, the Hessian is
$$\nabla^2 f(x,y) = \begin{bmatrix} -2x & 0 \\ 0 & -2 \end{bmatrix}.$$
The eigenvalues of $\nabla^2 f(-1,0)$ are $2$ and $-2$. Thus, $\nabla^2 f(-1,0)$ is indefinite and $(-1,0)$ is a saddle point. The eigenvalues of $\nabla^2 f(1,0)$ are $-2$ and $-2$. Thus, $\nabla^2 f(1,0)$ is negative definite and $(1,0)$ is a maximum.
With
$f(x, y) = (x + y)(1 - xy) \tag 1$
and
$\nabla f = (f_x, f_y), \tag 2$
we see that
$f_x = 1 - xy + (x + y)(-y) = 1 - xy - xy - y^2 = 1 - 2xy - y^2, \tag 3$
and likewise
$f_y = 1 - xy + (x + y)(-x) = 1 - xy - x^2 - xy = 1 - 2xy - x^2; \tag 4$
at critical points of $f(x, y)$, we have
$\nabla f(x, y) = 0, \tag 5$
whence from (3) and (4), at the critical points,
$1 - 2xy - y^2 = 0 = 1 - 2xy - x^2; \tag 6$
we solve this system by observing that it implies
$y^2 = 1 - 2xy = x^2, \tag 7$
so that
$y = \pm x; \tag 8$
using (8) in (6) we may write an equation for $x$:
$0 = x^2 + 2xy - 1 = x^2 \pm 2x^2 - 1; \tag 9$
$x$ must thus obey
$3x^2 - 1 = 0 \tag{10}$
or
$x^2 + 1 = 0; \tag{11}$
we rule out (11) since $x$ is real; thus
$x = \pm \dfrac{1}{\sqrt 3} = \pm \dfrac{\sqrt 3}{3}; \tag{12}$
again from (6),
$y = \dfrac{1 - x^2}{2x} = \dfrac{\dfrac{2}{3}}{2x} = \dfrac{1}{3x}; \tag{13}$
therefore the critical points are
$(x, y) = \left ( \dfrac{\sqrt 3}{3}, \dfrac{\sqrt 3}{3} \right ), \; (x, y) = \left ( -\dfrac{\sqrt 3}{3}, -\dfrac{\sqrt 3}{3} \right ); \tag{14}$
the Hessian $H_f$ of $f(x, y)$ has been provided for us courtesy of our OP kronos:
$H_f = \begin{bmatrix} -2y & -2x-2y \\ -2x-2y & -2x \end{bmatrix}; \tag{15}$
we thus see that
$\det(H_f) = 4xy - 4(x + y)^2 = 4(xy - (x + y)^2) = -4(x^2 +xy + y^2); \tag{16}$
since at the critical points we have
$x = y = \pm \dfrac{\sqrt 3}{3}, \tag{17}$
it follows that at these points
$\det(H_f) = -4, \tag{18}$
which, as is well-known, implies that each critical point is a saddle.
Best Answer
One way is to find the Hessian and determine its curvature by looking at its eigenvalues.
If the eigenvalues are all positive, then the function as positive curvature at that point, and you've found a minima.
If the eigenvalues are all negative, then negative curvature. And if they're mixed, then it's a saddle point.