So you came up with a formula which is (with very little rearrangement)
$$V = \frac{(xyz)^2}{2}+2 xyz+\frac{(xyz)^3}{24}+\frac83,$$
and you know from the very beginning that
$$xyz = 2.$$
Having those two pieces of information, you are a very short step away from
the solution. You should be able to use a simple substitution to find the
numeric value of the first formula.
The second formula tells you a substitution you can use.
I don't know exactly how you got the first formula; a comparison to a more
general formula for $xyz = c,$ where $c$ is an arbitrary positive constant,
makes me think you already used the specific fact that $xyz = 2.$
(Which is perfectly OK, because you were given that fact in the problem statement.)
Since the resulting volume for $xyz = 2$ agrees with the general solution,
however, I don't see any reason to suspect that you made any errors.
You can also look at the solutions and links mentioned in
Show that product of x, y, and z intercents of tangent plane to surface xyz=1 is a constant.
Your question is almost a duplicate of that one (up to a constant factor),
except that you had gotten much closer to a solution
at the point where you asked your question.
Here is an illustration of the problem at hand.
You want to find the the volume of the tetrahedron formed by the $x=y=z=0$ planes, and the plane tangent to the ellipsoid at some point on its surface, $(x,y,z)$. This volume, of course, will vary with different points on the surface of the ellipse, so the constraint function must be the following equality
$$\left(\frac{x}{a}\right)^2+\left(\frac{y}{b}\right)^2+\left(\frac{z}{c}\right)^2=1$$
Now the vector normal to the surface of the ellipsoid at some point $(x,y,z)$ is given by the gradient of its equation.
$$\nabla g (x,y,z) = \left<\frac{2x}{a^2},\frac{2y}{b^2},\frac{2z}{c^2}\right>$$
The equation for the plane is tangent to the ellipsoid at this point is easily worked out to be
$$\frac{x}{a^2}x'+\frac{y}{b^2}b'+\frac{z}{c^2}z'=1$$
Now, finally, for the volume that this plane bounds within the first quadrant, we make use of the fact that this volume is simply $1/6$ the volume of the rectangular prism whose side lengths are given by the points of where the plane intersects the three axes.
$$V=\frac{(abc)^2}{6xyz}$$
And that is your function $f$ which you are trying to minimize.
$$f=V$$
$$g\rightarrow \left(\frac{x}{a}\right)^2+\left(\frac{y}{b}\right)^2+\left(\frac{z}{c}\right)^2=1$$
Best Answer
Assume for the moment $a=b=c=1$. The surface $\xi$ then is the unit sphere $S^2$. A typical point ${\bf n}$ of $S^2$ in the first octant then has coordinates $(n_1,n_2,n_3)$ with $n_i>0$ and $\sum_i n_i^2=1$. Furthermore the tangent plane to $S^2$ at ${\bf n}$ is given by the equation ${\bf n}\cdot{\bf x}=1$, i.e., $n_1x_1+n_2x_2+n_3x_3=1$. Intersecting this plane with the three coordinate axes gives the points $\bigl({1\over n_1},0,0\bigr)$, $\bigl(0,{1\over n_2},0\bigr)$, $\bigl(0,0,{1\over n_3}\bigr)$. It follows that the simplex $S$ in question has volume $${\rm vol}(S)={1\over 6 n_1n_2n_3}\ .$$ This volume is minimal when its reciprocal is maximal. By the AMG inequality $$\root3\of {n_1^2 n_2^2n_3^2}\leq{1\over3}(n_1^2+n_2^2+n_3^2)={1\over3}\ ,$$ with equality iff $n_i=3^{-1/2}$ for all $i\in[3]$. It follows that $${\rm vol}(S)\geq{1\over6} 3^{3/2}={\sqrt{3}\over2}\ .$$ In the case of arbitrary $a$, $b$, $c>0$ we therefore have $${\rm vol}(S)\geq{\sqrt{3}\over2}abc\ ,$$ by standard geometric principles.