1) You really should have simplified your conditions first by dividing out 2 and 4 before differentiating it. This does not change the result, but it is needless complication.
2) $x$, $y$ and $z$ are positive numbers, so we have to take into account the behaviour on the border when one or more of them are zero.
The situation is symmetric, so say $z=0$, but then you have $x+y=50$ and $xy= 750$ which gives (by the solution formula of the quadratic equation) that $x$ and $y$ are not real numbers.
This means that subject to the two conditions there is actually no border and every extrema can be found by using Lagrange multipliers.
3)
You write:
$x(y−z)/2=λ(y−z)$, so that $λ=x/2$
But what you get, for each of the three similar equations, is:
Either (a) $y=z$ or (b) $\lambda=x/2$.
Either (a) $z=x$ or (b) $\lambda=y/2$.
Either (a) $x=y$ or (b) $\lambda=z/2$.
Now, you would have to choose (a) or (b) for each of the three equations which gives 8 cases (aaa,aab,aba,abb,baa,bab,bba,bbb), but it is simpler than that.
If you choose at least once condition (a), then you have two equal variables. If you choose at least twice condition (b), then you have two variables equal to $2\lambda$, so you also have two equal variables.
So, whatever you do, two variables will be equal.
But the problem statement is symmetric in $x$, $y$ and $z$ (which means that if you interchange the three variables, the problem statement does not change, for example $yzx=xyz$, so the volume you are looking for will not depend on the order of the variables).
Therefore, since you know that two variables are equal and you know that it doesn't really matter for the end result which ones are equal, it suffices to investigate $x=y$:
(At this point, we still might have $x=y=z$, but we don't know that. What we do know, is that the original conditions are true, so we use them first, and it turns out that they already fix all the values.)
The original conditions give:
$2x+z=50$ and $x^2+2xz=750$.
Substituting $z=50-2x$ in the second equation gives a quadratic equation $3x^2-100x+750=0$.
This gives $x=\frac{50\pm 5\sqrt{10}}{3}$.
You find two solutions (up to symmetry):
$x=y=\frac53 (10 + \sqrt{10})$, $z=\frac{10}3(5-\sqrt{10})$ giving the volume
$\frac{2500}{27}(35-\sqrt{10})\approx 2948$
and
$x=y=\frac 53 (10- \sqrt{10})$, $z=\frac{10}3(5+\sqrt{10})$ giving the volume
$\frac{2500}{27}(35+\sqrt{10})\approx 3534$.
Since these are the only two candidates for extrema, the first one is the minimum value and the second one is the maximum value.
If you solve the first three equations of the system$$\left\{\begin{array}{l}2(x-1)+2\lambda=0\\2(y-1)+\lambda=0\\2(z-1)-\lambda=0\\2x+y-z=5,\end{array}\right.$$then you get that $x=1-\lambda$, that $y=1-\frac\lambda2$, and that $z=1+\frac\lambda2$. But then$$5=2x+y-z=2-3\lambda,$$and therefore $\lambda=-1$. And so $(x,y,z)=\left(2,\frac32,\frac12\right)$. Therefore, the shortest distance is\begin{align}\left\|(1,1,1)-\left(2,\frac32,\frac12\right)\right\|&=\left\|\left(-1,-\frac12,\frac12\right)\right\|\\&=\frac12\|(-2,-1,1)\|\\&=\frac{\sqrt6}2.\end{align}
Best Answer
The formula you have for the volume ($V=xyz$) is the volume of the portion of the box in the first octant. But there are 8 octants, each symmetric to the first, so the volume should really be calculated as
$$V=8xyz$$
Updating your solution with the new objective function, it is clear that $x,y,z$ do not change--only the lagrange multiplier $\lambda$ will be 8 times its previous value.
Plugging in the $x,y,z$ you found, we get that the volume is $\frac{16\sqrt{6}}{3}$, as the given solutions indicate.
To help with the intuition surrounding this problem, we can reduce it to the 2D case. Consider the (poorly drawn) image below, and let $(x,y)$ be the coordinates of the upper right corner of the red box.
Then you have that the area of the red box is $A=xy$, but the acutal area of the insctibed rectangle is the orange box, given by $A=4xy$, doubling once for each dimension.
In the 3D case, there are 3 dimensions in which the volume needs to be doubled, but the concept is the same.