There is already an accepted answer, but I thought I'd leave some remarks since this is sort of a curious constraint surface. The function $ \ f(x,y,z) \ = \ x^2 + y^2 + z^2 \ $ can of course be thought of as the squared-distance from the origin to a point on the surface $ \ x^3 + y^3 - z^3 \ = \ 3 \ $ . Since this is an "open" surface, the function $ \ f \ $ has no absolute maximum value. The discussion by user112018 and mathlove shows that the choice of coordinate value for the location of extrema then are either zero or given by
$$ \ \lambda \ = \ \frac{2}{3x} \ = \ \frac{2}{3y} \ = \ -\frac{2}{3z} \ \ \Rightarrow \ \ x \ = \ y \ = \ -z \ . $$
As mathlove has said, this offers eight possibilities, less one, since $ \ ( 0, 0, 0) \ $ is not on the surface. The three remaining categories are:
I -- two coordinates are zero
$$ -z^3 \ = \ 3 \ \ \Rightarrow \ \ (0, 0, -3^{1/3}) \ , \ y^3 \ = \ 3 \ \ \Rightarrow \ \ (0, 3^{1/3}, 0) \ , \ x^3 \ = \ 3 \ \ \Rightarrow \ \ (3^{1/3}, 0, 0) $$
$$ \Rightarrow \ \ f \ = \ 3^{2/3} $$
for all three points. These are found by following along each coordinate axis until it intersects the "dimple".
II -- one coordinate is zero
$$ y^3 \ - \ z^3 \ = \ 3 \ , \ y \ = \ -z \ \Rightarrow \ \ 2y^3 \ = \ 3 \ \ \Rightarrow \ \ (0, \left( \frac{3}{2} \right)^{1/3} , -\left( \frac{3}{2} \right)^{1/3} ) \ ,$$
$$ x^3 \ - \ z^3 \ = \ 3 \ , \ x \ = \ -z \ \ \Rightarrow \ \ ( \left( \frac{3}{2} \right)^{1/3} , 0 , -\left( \frac{3}{2} \right)^{1/3} ) \ ,$$
$$ x^3 \ + \ y^3 \ = \ 3 \ , \ x \ = \ y \ \Rightarrow \ \ 2x^3 \ = \ 3 \ \ \Rightarrow \ \ (\left( \frac{3}{2} \right)^{1/3} , \left( \frac{3}{2} \right)^{1/3} , 0 ) $$
$$ \Rightarrow \ \ f \ = \ 2 \cdot \left( \frac{3}{2} \right)^{2/3} \ = \ 2^{1/3} \cdot 3^{2/3} $$
for all three points. These are located along diagonals, in each of the three coordinate planes, meeting the "dimple" .
III -- no coordinate is zero
$$ x \ = \ y \ = \ -z \ \Rightarrow \ \ 3x^3 \ = \ 3 \ \ \Rightarrow \ \ (1, 1 , -1 ) \ \ \Rightarrow \ \ f \ = \ 3 \ . $$
This point lies on a "body diagonal" running to the "deepest corner of the dimple".
So, owing to the particular symmetries and anti-symmetries of this surface, the squared-distance function has three local and absolute minima [Category I] and what appear to be three additional (shallow) local minima [Category II] and a (very shallow) local maximum [Category III].
Best Answer
$f(x,y) = xy^2$
With condition $x^2 + y^2 \leq 3$
Based on our function $(f(x,y) = xy^2)$, one thing to note is that our extrema would likely occur on the boundary $x^2 + y^2 = 3$.
Continuing from the point that you got to in your question,
$y^2=2x\lambda \implies \lambda = \frac{y^2}{2x} \, $ if $x \ne 0$
$2xy=2y\lambda \implies \lambda = x \,$ or $ \,x = 0$
For $x = 0, y = 0$
For $x \ne 0$, equating $\lambda, \, y = \pm \sqrt2 \, x$
Based on the condition, $-1 \leq x \leq 1$
So potential points to check for min and max are $(-1, - \sqrt2), (-1, \sqrt 2), (1, -\sqrt2), (1, \sqrt 2), (0, 0)$.