Well, we have $f(x,y)=x^2y^2-2x-2y$ considered on the following green area:
So $f_x=2xy^2-2,~~f_y=2x^2y-2$ and solving $f_x=f_y=0$ gives us the critical point $x=y=1$ which is on the border of the area. Now think about the border:
$$y=5,~~ 0\le x\le 5 \;\;\;\; x=0,~~0\le y\le 5 \;\;\;\; y=x,~~ 0\le x\le 5$$
If $y=5,~~ 0\le x\le 5$, so $$f(x,y)\to f(x)=25x^2-2x-10$$
and this easy for you to find the critical points which are maybe the best candidates on this border.
If $x=0,~~0\le y\le 5$, so $$f(x,y)\to f(y)=-2y$$ there is a similar way of finding such points on this border.
You should think geometrically. The function $$f(x,y)=-x^{2012}-y^{2012}$$ can be written $$f(x,y)=-(x^{2012}+y^{2012})$$ But because 2012 is even, $x^{2012}\geq0$ and $y^{2012}\geq 0$. Therefore $x^{2012}+y^{2012}\geq0$ and so $f(x,y)\leq 0$ for all points $(x,y)$. Therefore your critical point $(0,0)$ is a global maximum.
Whoever assigned this problem probably wanted you to notice that your function $f$ is similar to
$$g(x,y)=x^2+y^2$$which is an upward-opening elliptic paraboloid that has a global minimum at the origin. Similarly, $$h(x,y)=-x^2-y^2$$ is a downward-opening elliptic paraboloid that has a global maximum at the origin.
The only difference between these functions and your function $f$ is that $f$ rises much, much faster when $|x|>1$ and $|y|>1$ and is much, much flatter when $|x|<1$ and $|y|<1$, because the exponent is so much higher. In fact, the graph would look more like a box than a paraboloid because the walls of the surface are so steep.
In general, there's no surefire method for analyzing the local behavior of functions where the second derivative test comes back inconclusive. In practice, you should think geometrically or look at higher order derivatives to get a sense of what's going on.
To use the latter approach, consider taking the 2012th partial derivatives of your function. Both the $x$ and $y$ partial at the origin will be $-2012!$, and combining this with the multivariate Taylor expansion (and reflecting on the mixed partials) gives another way of seeing that your function is negative definite near the origin. (The reason the second derivative test fails for this function is that it is too flat near its critical point. This extreme flatness is what makes so many of the higher-order derivatives zero.)
But your function is so simple to understand that its global properties are obvious if you think geometrically. Other functions might require you to think harder about what's happening graphically, in which case the analytic approach with higher derivatives could be helpful.
Best Answer
Clearly, $\nabla f(x,y)=0\iff(x,y)=(0,0)$, but $(0,0)$ is a saddle-point. So, the maximum and the minimum can be attained only at the boundary of $D$.
Let $h(x,y)=x^2+\frac{y^2}4$. Then apply the method of Lagrange multipliers:$$\left\{\begin{array}{l}f_x(x,y)=\lambda h_x(x,y)\\ f_y(x,y)=\lambda h_y(x,y)\\h(x,y)=1,\end{array}\right.$$or$$\left\{\begin{array}{l}2x=2\lambda x\\-2y=\frac12\lambda y\\x^2+\frac{y^2}4=1.\end{array}\right.$$It's solutions are $(x,y,\lambda)=(0,-2,-4)$, $(x,y,\lambda)=(0,2,-4)$, $(x,y,\lambda)=(-1,0,1)$ and $(x,y,\lambda)=(1,0,1)$. Since $f(0,\pm2)=-2$, and $f(\pm1,0)=3$, the maximum is $3$ and the minimum is $-2$.