I think that the method of Lagrange multipliers is the easiest way to
solve my question, but how can I find the Lagrangian function?
As shown by other answers and in note 1 there are easier ways to find the shortest distance, but here is a detailed solution using the method of Lagrange multipliers. You need to find the minimum of the distance function
$$\begin{equation}
d(x,y,z)=\sqrt{x^{2}+y^{2}+(z-1)^{2}} \tag{1a}
\end{equation}$$
subject to the constraint given by the surface equation $z=f(x,y)=\frac32(x^2+y^2)$
$$\begin{equation}
g(x,y,z)=z-\frac{3}{2}\left( x^{2}+y^{2}\right) =0. \tag{2}
\end{equation}$$
Since $\sqrt{x^{2}+y^{2}+(z-1)^{2}}$ increases with $x^{2}+y^{2}+(z-1)^{2}$ you can simplify the
computations if you find the minimum of
$$\begin{equation}
[d(x,y,z)]^2=x^{2}+y^{2}+(z-1)^{2} \tag{1b}
\end{equation}$$
subject to the same constraint $(2)$. The Lagrangian function is then
defined by
$$\begin{eqnarray}
L\left( x,y,z,\lambda \right) &=&[d(x,y,z)]^2+\lambda g(x,y,z) \\
L\left( x,y,z,\lambda \right) &=&x^{2}+y^{2}+(z-1)^{2}+\lambda \left( z-
\frac{3}{2}\left( x^{2}+y^{2}\right) \right), \tag{3}
\end{eqnarray}$$
where $\lambda $ is the Lagrange multiplier. By this method you need to
solve the following system
$$\begin{equation}
\left\{ \frac{\partial L}{\partial x}=0,\frac{\partial L}{\partial y}=0,
\frac{\partial L}{\partial z}=0,\frac{\partial L}{\partial \lambda }
=0,\right. \tag{4}
\end{equation}$$
which results in
$$\begin{eqnarray}
\left\{
\begin{array}{c}
2x+3\lambda x=0 \\
2y+3\lambda y=0 \\
2z-2-\lambda =0 \\
-z+\frac{3}{2}\left( x^{2}+y^{2}\right) =0
\end{array}
\right. &\Leftrightarrow &\left\{
\begin{array}{c}
x=0\vee 2+3\lambda =0 \\
y=0\vee 2+3\lambda =0 \\
2z-2-\lambda =0 \\
-z+\frac{3}{2}\left( x^{2}+y^{2}\right) =0
\end{array}
\right. \\
&\Leftrightarrow &\left\{
\begin{array}{c}
x=0 \\
y=0 \\
\lambda =2 \\
z=0
\end{array}
\right. \vee \left\{
\begin{array}{c}
\lambda =-2/3 \\
z=2/3 \\
x^{2}+y^{2}=4/9
\end{array}
\right. \tag{5}
\end{eqnarray}$$
For $x=y=x=0$ we get $d(0,0,0)=1$. And for $x^2+y^2=4/9,z=2/3$ we get the minimum distance subject to the given conditions
$$\begin{equation}
\underset{g(x,y,z)=0}\min d(x,y,z)=\sqrt{\frac{4}{9}+(\frac{2}{3}-1)^{2}}=\frac{1}{3}\sqrt{5}. \tag{6}
\end{equation}$$
It is attained on the intersection of the surface $z=\frac{3}{2}\left(
x^{2}+y^{2}\right) $ with the vertical cylinder $x^{2}+y^{2}=\frac{4}{9}$ or equivalently with the horizontal plane $z=\frac{2}{3}$.
$$\text{Plane }z=\frac{2}{3} \text{(blue) and surface }z=\frac{3}{2}\left(
x^{2}+y^{2}\right) $$
Notes.
- As the solution depends only on the sum $r^{2}=x^{2}+y^{2}$ we could just find
$$\begin{equation}
\min [d(r)]^2=r^{2}+(\frac{3}{2}r^{2}-1)^{2} \tag{7}
\end{equation}$$
and then find $d(r)=\sqrt{[d(r)]^2}$ at the minimum.
- The surface $z=\frac{3}{2}\left(
x^{2}+y^{2}\right) $ is a surface of revolution around the $z$ axis.
No one can be zero. So, in that surface, $z^2=2/xy$. Now, by AM-GM
$$x^2+y^2+z^2=x^2+y^2+\frac{2}{xy}\geq2xy +\frac{2}{xy}\geq 2\sqrt{4}=4.$$ Study the conditions for the equality to happen and show that they actually happen for the points you already suspect.
Best Answer
I think I can provide a simpler method for this specific problem that does not use Lagrange multipliers.
Take a cross section of the 3-dimensional solid $z=x^2+3y^2$ at the plane $y=0$. We then have a parabola on the $xz$ plane: $z=x^2, y=0$. Note that point $P$ is also on this plane. To find the smallest distance between the original surface and the point, all we have to do is find the smallest distance between the point and the parabola. We can do this by looking at the normal lines of the parabola, which are of the form $z-m^2=-\frac{1}{2m}(x-m), y=0$ for some point $(m,0,m^2)$. We want this line to contain the point $(5,0,1)$.
$$1-m^2=-\frac{1}{2m}(5-m)$$ $$2m-2m^3=-5+m$$ $$2m^3-m-5=0$$ $$m=\frac{\sqrt[3]{45-\sqrt{2019}}+\sqrt[3]{45+\sqrt{2019}}}{6^{2/3}}$$
Ergo, the distance we are looking for is the distance between the points $$\left(\frac{\sqrt[3]{45-\sqrt{2019}}+\sqrt[3]{45+\sqrt{2019}}}{6^{2/3}},0,\left(\frac{\sqrt[3]{45-\sqrt{2019}}+\sqrt[3]{45+\sqrt{2019}}}{6^{2/3}}\right)^2\right) \text{and } (5,0,1)$$
I know this may seem way off, but Behrouz Maleki's solution also agrees with this answer (look at case 1 of his answer).