With
$f(x, y) = (x + y)(1 - xy) \tag 1$
and
$\nabla f = (f_x, f_y), \tag 2$
we see that
$f_x = 1 - xy + (x + y)(-y) = 1 - xy - xy - y^2 = 1 - 2xy - y^2, \tag 3$
and likewise
$f_y = 1 - xy + (x + y)(-x) = 1 - xy - x^2 - xy = 1 - 2xy - x^2; \tag 4$
at critical points of $f(x, y)$, we have
$\nabla f(x, y) = 0, \tag 5$
whence from (3) and (4), at the critical points,
$1 - 2xy - y^2 = 0 = 1 - 2xy - x^2; \tag 6$
we solve this system by observing that it implies
$y^2 = 1 - 2xy = x^2, \tag 7$
so that
$y = \pm x; \tag 8$
using (8) in (6) we may write an equation for $x$:
$0 = x^2 + 2xy - 1 = x^2 \pm 2x^2 - 1; \tag 9$
$x$ must thus obey
$3x^2 - 1 = 0 \tag{10}$
or
$x^2 + 1 = 0; \tag{11}$
we rule out (11) since $x$ is real; thus
$x = \pm \dfrac{1}{\sqrt 3} = \pm \dfrac{\sqrt 3}{3}; \tag{12}$
again from (6),
$y = \dfrac{1 - x^2}{2x} = \dfrac{\dfrac{2}{3}}{2x} = \dfrac{1}{3x}; \tag{13}$
therefore the critical points are
$(x, y) = \left ( \dfrac{\sqrt 3}{3}, \dfrac{\sqrt 3}{3} \right ), \; (x, y) = \left ( -\dfrac{\sqrt 3}{3}, -\dfrac{\sqrt 3}{3} \right ); \tag{14}$
the Hessian $H_f$ of $f(x, y)$ has been provided for us courtesy of our OP kronos:
$H_f = \begin{bmatrix} -2y & -2x-2y \\ -2x-2y & -2x \end{bmatrix}; \tag{15}$
we thus see that
$\det(H_f) = 4xy - 4(x + y)^2 = 4(xy - (x + y)^2) = -4(x^2 +xy + y^2); \tag{16}$
since at the critical points we have
$x = y = \pm \dfrac{\sqrt 3}{3}, \tag{17}$
it follows that at these points
$\det(H_f) = -4, \tag{18}$
which, as is well-known, implies that each critical point is a saddle.
Best Answer
Careful, you have a small sign and exponent mistake: $\left(2x-2y , -2x \color{green}{+} 12y^\color{green}{2}\right)^T$.
But you haven't found any critical points yet, as you only used one equation so far. Keep in mind that requiring that the gradient is equal to $(0,0)$ is equivalent to a system of two equations: $$\left\{ \begin{array}{rcl} 2x-2y &=& 0 \\ -2x + 12y^2 &=& 0 \end{array}\right. \iff \left\{ \begin{array}{rcl} x &=& y \\ -2x + 12y^2 &=& 0 \end{array}\right.$$ Now plug $x=y$ into the second equation and solve: $$-2x + 12x^2 = 0 \iff -2 x\left( 1 -6x \right) = 0 \iff x = \ldots$$ Since $y=x$, each solution $x$ will correspond to a critical point $(x,x)$. Can you finish?