There is already an accepted answer, but I thought I'd leave some remarks since this is sort of a curious constraint surface. The function $ \ f(x,y,z) \ = \ x^2 + y^2 + z^2 \ $ can of course be thought of as the squared-distance from the origin to a point on the surface $ \ x^3 + y^3 - z^3 \ = \ 3 \ $ . Since this is an "open" surface, the function $ \ f \ $ has no absolute maximum value. The discussion by user112018 and mathlove shows that the choice of coordinate value for the location of extrema then are either zero or given by
$$ \ \lambda \ = \ \frac{2}{3x} \ = \ \frac{2}{3y} \ = \ -\frac{2}{3z} \ \ \Rightarrow \ \ x \ = \ y \ = \ -z \ . $$
As mathlove has said, this offers eight possibilities, less one, since $ \ ( 0, 0, 0) \ $ is not on the surface. The three remaining categories are:
I -- two coordinates are zero
$$ -z^3 \ = \ 3 \ \ \Rightarrow \ \ (0, 0, -3^{1/3}) \ , \ y^3 \ = \ 3 \ \ \Rightarrow \ \ (0, 3^{1/3}, 0) \ , \ x^3 \ = \ 3 \ \ \Rightarrow \ \ (3^{1/3}, 0, 0) $$
$$ \Rightarrow \ \ f \ = \ 3^{2/3} $$
for all three points. These are found by following along each coordinate axis until it intersects the "dimple".
II -- one coordinate is zero
$$ y^3 \ - \ z^3 \ = \ 3 \ , \ y \ = \ -z \ \Rightarrow \ \ 2y^3 \ = \ 3 \ \ \Rightarrow \ \ (0, \left( \frac{3}{2} \right)^{1/3} , -\left( \frac{3}{2} \right)^{1/3} ) \ ,$$
$$ x^3 \ - \ z^3 \ = \ 3 \ , \ x \ = \ -z \ \ \Rightarrow \ \ ( \left( \frac{3}{2} \right)^{1/3} , 0 , -\left( \frac{3}{2} \right)^{1/3} ) \ ,$$
$$ x^3 \ + \ y^3 \ = \ 3 \ , \ x \ = \ y \ \Rightarrow \ \ 2x^3 \ = \ 3 \ \ \Rightarrow \ \ (\left( \frac{3}{2} \right)^{1/3} , \left( \frac{3}{2} \right)^{1/3} , 0 ) $$
$$ \Rightarrow \ \ f \ = \ 2 \cdot \left( \frac{3}{2} \right)^{2/3} \ = \ 2^{1/3} \cdot 3^{2/3} $$
for all three points. These are located along diagonals, in each of the three coordinate planes, meeting the "dimple" .
III -- no coordinate is zero
$$ x \ = \ y \ = \ -z \ \Rightarrow \ \ 3x^3 \ = \ 3 \ \ \Rightarrow \ \ (1, 1 , -1 ) \ \ \Rightarrow \ \ f \ = \ 3 \ . $$
This point lies on a "body diagonal" running to the "deepest corner of the dimple".
So, owing to the particular symmetries and anti-symmetries of this surface, the squared-distance function has three local and absolute minima [Category I] and what appear to be three additional (shallow) local minima [Category II] and a (very shallow) local maximum [Category III].
It is a closed region, so max and min must occur. They can only occur on the boundary or at critical points of the function. So you can use the following steps:
Step 1: Find all the critical points of the function, and check whether they are in the constraint region.
Step 2: Use regular Lagrange multiplier method on the boundary of the disk.
Then combine the results from the two steps to find the max and min.
Best Answer
We have $y=x-4$ and $z=8-2y-x=16-3x$. This gives (show it !):
$f(x,y,z)=11x^2-104x+272$