You are substituting incorrectly. Here is the right way for $f(x,y)=2x^3 + 6xy^2 - 3y^3 - 150x$:
$$\begin{cases}f_x=6x^2+6y^2-150=0\\
f_y=12xy-9y^2=0\end{cases} \Rightarrow \begin{cases}x^2+y^2=25\\ y(4x-3y)=0\end{cases} \Rightarrow \\
1) \ \ y=0 \Rightarrow x^2+0^2=25 \Rightarrow x=\pm5;\\
2) \ \ 4x-3y=0 \Rightarrow x=\frac34y \Rightarrow \left(\frac34y\right)^2+y^2=25 \Rightarrow y^2=16 \Rightarrow y=\pm 4 \Rightarrow x=\pm3.$$
Hence, the stationary points are:
$$(x,y)=(5,0), (-5,0), (3,4), (-3,-4).$$
Note that once you set first order derivatives equal to zero, you must solve the system of equations to find $(x,y)$.
You are given that
$$g(x,y)=\frac{x^3}{3}+y^2-2xy-3y$$
So, the partial derivatives are
$$g_x=x^2-2y$$
$$g_{xx}=2x$$
$$g_y=2y-2x-3$$
$$g_{yy}=2$$
$$g_{xy}=-2$$
If we set $g_x=0$ then
$$g_x=x^2-2y=0 \implies x^2=2y$$
where
$$x=\pm \sqrt{2y}\tag{1}$$
$$y = \frac{x^2}{2}\tag{2}$$
and if we set $g_y=0$ then
$$g_y=2y-2x-3=0 \implies 2(y-x)=3 \implies y-x = \frac{3}{2}$$
where
$$y=x+\frac{3}{2}\tag{3}$$
$$x=y-\frac{3}{2}\tag{4}$$
So, equating (1) and (4) forms
$$y-\frac{3}{2}=\pm\sqrt{2y}$$
which can be reduced to
$$y=\frac{9}{2},~~~y=\frac{1}{2}$$
which substituting back into $(2)$ forms
$$\frac{9}{2}=\frac{x^2}{2} \implies x=3,-3$$
$$\frac{1}{2}=\frac{x^2}{2} \implies x=1,-1$$
which gives us the critical points $(-1,\frac{1}{2}), (1,\frac{1}{2}),(3,\frac{9}{2})$ and $(-3,\frac{9}{2})$.
We now need to apply the second partial derivative test. The test tells us that provided $g(x, y)$ is a differentiable real function of two variables whose second partial derivatives exist we can compute
$$D(x,y)=g_{xx}(x,y)g_{yy}(x,y)-\Big(g_{xy}(x,y)\Big)^2$$
and that
- If $D(a,b)>0$ and $g_{xx}(a,b)>0$ then $(a,b)$ is a local minimum of $g$.
- If $D(a,b)>0$ and $g_{xx}(a,b)<0$ then $(a,b)$ is a local maximum of $g$.
- If $D(a,b)<0$ then $(a,b)$ is a saddle point of $g$.
- If $D(a,b)=0$ then the second derivative test is inconclusive, and the point $(a, b)$ could be any of a minimum, maximum or saddle point.
We found before that
$$g_{xx}=2x$$
$$g_{yy}=2$$
$$g_{xy}=-2$$
Thus, if we compute $D(x,y)$ for our four critical points we have
$$D\big(-1,\frac{1}{2}\big)=(2\times -1)(2)-(-2)^2=-8 <0$$
$$D\big(1,\frac{1}{2}\big)=(2\times 1)(2)-(-2)^2=0$$
$$D\big(3,\frac{9}{2}\big)=(2\times 3)(2)-(-2)^2=8>0$$
$$D\big(-3,\frac{9}{2}\big)=(2\times -3)(2)-(-2)^2=-16<0$$
Therefore, by the second partial derivative test,
- $(-1,\frac{1}{2})$ is a saddle point by definition.
- $(1,\frac{1}{2})$ is inconclusive.
- $(3,\frac{9}{2})$ is a local minimum since $D\big(3,\frac{9}{2}\big) > 0$ and $g_{xx}(3,\frac{9}{2})=6>0$.
- $(-3,\frac{9}{2})$ is a saddle point by definition.
To get the value of $-\frac{45}{4}$, evaluate $g(x,y)$ at $(x,y)=(3,\frac{9}{2})$ to form
$$g\big(3,\frac{9}{2}\big)=\frac{3^3}{3}+\big(\frac{9}{2}\big)^2 - 2(3)\big(\frac{9}{2}\big)-3\big(\frac{9}{2}\big)=-\frac{45}{4}$$
which is the local minimum that you found through Wolfram Alpha. Similarly, to get $\frac{-7}{12}$ evaluate $g(x,y)$ at $(x,y)=(-1,\frac{1}{2})$ to form
$$g\big(-1,\frac{1}{2}\big)=\frac{(-1)^3}{3}+\big(\frac{1}{2}\big)^2 - 2(-1)\big(\frac{1}{2}\big)-3\big(\frac{1}{2}\big)=-\frac{7}{12}$$
which is a saddle point.
Best Answer
The derivatives are:
${\partial z \over \partial x} = 3 x^2 - 6 y$
${\partial z \over \partial y} = 3 y^2 - 6 x$
Set the pair to $0$ to find the two (real) solutions: $(2,2)$ and $(0,0)$.
Take the second derivatives evaluated at those points to find a minimum at $(2,2)$ and saddle at $(0,0)$, as seen in the figure.
${\partial^2 z \over \partial x^2} = 6 x$
${\partial^2 z \over \partial y^2} = 6 y$