Start with the equations that you have derived:
\begin{eqnarray*}
ye^{xy}&=&3\lambda x^2,\\
xe^{xy}&=&3\lambda y^2,\\
x^3+y^3&=&16.
\end{eqnarray*}
As a first step, show that none of $\lambda, x$ or $y$ can be zero. (If one of them is zero, then the first two equations show that all three must be zero, contradicting the third equation.) This is useful as we now know that we can divide by these terms at will.
Now comparing the first and second equations gives $x^3=y^3$, and since both are real, we get $x=y$. The third equation then gives $x=y=2$, and the first or second yields the value of $\lambda$. We find $f=e^4$ at this point, and it must be a maximum.
As regards the minimum, recall that the Lagrange multiplier method identifies the possible location of max/min points IF they exist. I don't think your example has a minimum: By taking $x^3 = N^3$ and $y^3=16-N^3$, where $N$ is a large positive number, the constraint $x^3+y^3=16$ is satisfied. But $xy\sim-N^2$ can be made arbitrarily large and negative, so that $e^{xy}$ can be made as close as we like to $0$, but of course would never equal zero. So we can say that the infimum
$$\inf\{e^{xy}:x^3+y^3=16\}=0,$$
but the minimum
$$\min\{e^{xy}:x^3+y^3=16\}$$
does not exist: for any candidate minimum at $(x_0,y_0)$, we can always find $(x_1,y_1)$ with $0<f(x_1,y_1)<f(x_0,y_0)$.
Another approach is to eliminate $y$ and to treat this as a single variable max/min problem for $h(x)=\exp[x(16-x^3)^{1/3}]$. This gives another way of understanding why there is no minimum point of the function.
I do not see where you used Lagrange multipliers. Any critical points will satisfy the Lagrange multipliers equation
$$
\begin{bmatrix}
yz & xz & xy
\end{bmatrix} = \lambda \begin{bmatrix}
2x & 4y & 6z
\end{bmatrix} \ .
$$
This gives you the system of equations
$$
\begin{align}
yz &= 2\lambda x & (1) \\
xz &= 4\lambda y & (2) \\
xy &= 6\lambda z & (3) \\
96 &= x^2+2y^2+3z^2 & (4)
\end{align}
$$
First consider $\lambda = 0$ and then consider other cases. Try this and see how it goes for you. I hope this helps.
EDIT:
If you assume $x,y,z\neq 0$, you can solve for $\lambda$ in each of $(1), (2), (3),$ and $(4)$. You then obtain (by equating these $\lambda$), as you did, $6y^2=96$, which gives $y=\pm4$. You similarly obtain $x^2 = 32$, or $x=\pm4\sqrt{2}$, and $z^2 = \frac{32}{3}$, or $z=\pm4\sqrt{\frac{2}{3}}$. With all of the $\pm$'s, you get a few critical points. Plug them in to see which ones are the largest/smallest.
but I'm stuck from this point, can anyone help me continue and find the min max vales?
Best Answer
We can re-write the first three equations as $$2x(1-\frac\lambda{\alpha^2})=0\\2y(1-\frac\lambda{\beta^2})=0\\2z(1-\frac\lambda{\gamma^2})=0$$ Notice that since $\alpha\ne\beta\ne\gamma$ at least two of the $x,y,z$ must be $0$. From the equation of the constraint you see that $x=y=z=0$ is not a solution. Therefore the only possible solutions are
The value of $f(x,y,z)$ in these points is $\gamma^2$, $\beta^2$, and $\alpha^2$ for the above cases. The maximum is the last option, the minimum is the first.