we can use the distance formula, but for the sake of simplicity we can use $f(x,y,z)=x^2+y^2+z^2$ and let $\phi(x,y,z)=x-yz=10$. Via Lagrange multiplier,
\begin{align}\partial_xf+\partial_x\phi\cdot\lambda=0\\
2x+\lambda=0\tag{i}\\
\partial_yf+\partial_y\phi\cdot\lambda=0\\
2y-z\lambda=0\tag{ii}\\
\partial_zf+\partial_z\phi\cdot\lambda=0\\
2z-\lambda y=0\tag{iii}
\end{align}
from equating (ii) and (iii) we find that $$2z=\frac{\lambda^2z}{2}\implies \lambda=\pm 2$$
for $\lambda=2,y=z,$ and $x=-1$, which means
$$-1-yz=10$$
$$-yz=11$$
but since $y=z$
$$y^2=-11$$
Therefore $\lambda=-2\implies y=-z,x=1$
\begin{align}
-y^2=-9\\
y=3\\
z=-3
\end{align}
Therefore the shortest distance from the origin to the plane is $\sqrt{9+9+1}=\sqrt{19}$
I believe this should be correct, but I have no idea how to verify it, could anyone help me confirm this, or point out where I made a mistake? Thanks!
Best Answer
Here is an alternative approach as a verification. The normal vector of the surface is $(-1,z,y)$. At the point of the shortest distance, the normal vector is parallel to $(x,y,z)$, the direction of the distance line, i.e.
$$\frac x{-1} = \frac yz = \frac zy$$
which leads to the point either at $(-1, y, y) $ or $(1, y, -y)$. Plug them into the surface equation to find that $(-1, y, y) $ is invalid and $(1, y, -y)$ yields $y^2=9$. As a result, the points with the shortest distance are,
$$(1, \pm 3, \mp3)$$
Thus, the shortest distance is $\sqrt{1+9+9}=\sqrt{19}$. Note, the result matches yours, except that it comes from two points on the surface.