Let's look at what happens when we perturb $(x,y,z)$ infinitesimally in the direction $(\delta x,\delta y,\delta z)$.
To stay on $x^2+y^2+z^2=1$, we need $x\,\delta x+y\,\delta y+z\,\delta z=0$ and to stay on $3x+y=3$, we need $3\delta x+\delta y=0$.
That is, we want to consider only $(\delta x,\delta y,\delta z)$ which are perpendicular to both $(x,y,z)$ and $(3,1,0)$.
We want to find the point at which $\delta f(x,y,z)=(0,1,1)\cdot(\delta x,\delta y,\delta z)=0$ for all $(\delta x,\delta y,\delta z)$ which are perpendicular to both $(x,y,z)$ and $(3,1,0)$. For that to be true, $(0,1,1)$ must be a linear combination of $(3,1,0)$ and $(x,y,z)$. This means that $(x,y,z)$ is also a linear combination of $(3,1,0)$ and $(0,1,1)$; that is, on the plane $x-3y+3z=0$.
Thus, we are looking for a point on the unit sphere that is on the planes $x-3y+3z=0$ and $3x+y=3$. The intersection of these two planes is the line $(0,3,3)+(-3,9,10)t$. So we need to solve
$$
\begin{align}
1&=(0-3t)^2+(3+9t)^2+(3+10t)^2\\
&=18+114t+190t^2
\end{align}
$$
that is $t=\dfrac{-57\pm\sqrt{19}}{190}$. Note that
$$
\begin{align}
f((0,3,3)+(-3,9,10)t)
&=(0,1,1)\cdot((0,3,3)+(-3,9,10)t)\\
&=6+19t
\end{align}
$$
Plugging in the values of $t$ give us the extreme values: $\dfrac{3+\sqrt{19}}{10}$ and $\dfrac{3-\sqrt{19}}{10}$
Best Answer
Using the first equation, $$x^2+y^2+z^2+2(xy+yz+zx)=100$$
From this equation, subtract the second equation, so we are left with, $$2(xy+yz+zx)=64\\(xy+yz+zx)=32$$
Thus let x,y,z be the roots of the cubic eq P(a)=0 ,using vieta's theorem
$$a^3−10a^2+32a−p=0$$ where $p=xyz$.
Now you can find the maxima and minima of $a^3−10a^2+32a$ and hence $p$. Apparently, the maxima and minima come out to be infinitely large. Why?
This is because, the domain of $a$ is such that we consider complex value of the variables named $(x,y,z)$. For the real values of the variables, it will only exist when $\dfrac{dp}{da}\leq 0$ (in other words we find the local extremas). Differentiating and equating to zero for $a^3−10a^2+32a$ gives us, $$3a^2−20a+32=0$$ and the roots $a=\frac{8}{3}, 4$. Which further yields the maximum and minimum for the real domain as $32$ and $33.185$. You can verify this yourself following this.