The volume of a simplex in $\mathbb{R}^n$ (tetrahedron in $\mathbb{R}^3$), one of whose vertices is the origin, is $\frac{1}{n!}$ times the determinant of the matrix with the other vertices as columns. Thus, the volume of your tetrahedron is
$$
\frac{1}{3!}\det\begin{bmatrix}1&0&0\\0&1&0\\0&0&1\end{bmatrix}=\frac16
$$
and $4\cdot\frac16=\frac23$, which is hopefully the correct answer.
The volume integral should be
$$
\int_0^1\int_0^{1-z}\int_0^{1-z-y}4\,\mathrm{d}x\,\mathrm{d}y\,\mathrm{d}z
$$
That is, $z$ can take on any value in $[0,1]$. The integral in $y$ is taken over each particular value of $z$, so $y$ can take any value in $[0,1-z]$. The integral in $x$ is taken over each particular value of $z$ and $y$, so $x$ can take any value in $[0,1-z-y]$.
On the surface $x+y+z=1$, the surface normal is $\nabla(x+y+z-1)=(1,1,1)$. Normalizing this gives the unit normal: $n=\frac{1}{\sqrt{3}}(1,1,1)$.
To integrate over this surface, the integral would be
$$
\begin{align}
&\int_0^1\int_0^{1-y}F\cdot n\sqrt{\left(\frac{\partial z}{\partial x}\right)^2+\left(\frac{\partial z}{\partial y}\right)^2+1}\;\mathrm{d}x\,\mathrm{d}y\\
&=\int_0^1\int_0^{1-y}F\cdot n\sqrt{3}\;\mathrm{d}x\,\mathrm{d}y\\
&=\int_0^1\int_0^{1-y}(4x,x^2y,-x^2z)\cdot(1,1,1)\frac{1}{\sqrt{3}}\sqrt{3}\;\mathrm{d}x\,\mathrm{d}y\\
&=\int_0^1\int_0^{1-y}(4x+x^2y-x^2z)\;\mathrm{d}x\,\mathrm{d}y\\
&=\int_0^1\int_0^{1-y}(4x+x^2y-x^2(1-x-y))\;\mathrm{d}x\,\mathrm{d}y\\
&=\int_0^1\int_0^{1-y}(x^3-x^2+4x+2x^2y)\;\mathrm{d}x\,\mathrm{d}y
\end{align}
$$
The other surfaces should be much simpler.
So you came up with a formula which is (with very little rearrangement)
$$V = \frac{(xyz)^2}{2}+2 xyz+\frac{(xyz)^3}{24}+\frac83,$$
and you know from the very beginning that
$$xyz = 2.$$
Having those two pieces of information, you are a very short step away from
the solution. You should be able to use a simple substitution to find the
numeric value of the first formula.
The second formula tells you a substitution you can use.
I don't know exactly how you got the first formula; a comparison to a more
general formula for $xyz = c,$ where $c$ is an arbitrary positive constant,
makes me think you already used the specific fact that $xyz = 2.$
(Which is perfectly OK, because you were given that fact in the problem statement.)
Since the resulting volume for $xyz = 2$ agrees with the general solution,
however, I don't see any reason to suspect that you made any errors.
You can also look at the solutions and links mentioned in
Show that product of x, y, and z intercents of tangent plane to surface xyz=1 is a constant.
Your question is almost a duplicate of that one (up to a constant factor),
except that you had gotten much closer to a solution
at the point where you asked your question.
Best Answer
I can't tell where you went wrong: perhaps you forgot to integrate along all the surface that bounds your volume $V$?
Namely, the divergence theorem says
$$ \int_S \langle F, dS\rangle = \int_V \mathrm{div}Fdxdydz , \ $$
where $S = \partial V = S_1 + S_2 + S_3 + S_4$, where $S_1$ is your plane $2x + 3y + 6z = 12$ and $S_2, S_3, S_4$ the (portions of the) coordinates planes that help $S_1$ bound $V$. That is, the integral on the left hand side is actually composed of four integrals:
$$ \int_S \langle F, dS\rangle = \int_{S_1} \langle F, dS \rangle + \dots + \int_{S_4} \langle F, dS \rangle \ . $$
EDIT. Ok, so a few more computations. On the one hand,
$$ \int_{S_1}\langle F, dS \rangle = \int_{D_1} \langle F\circ\varphi, \varphi_x \times \varphi_y\rangle dxdy \ , $$
where $D_1 = \left\{ 0\leq x \leq 6, 0 \leq y \leq \frac{12 -2x}{3}\right\}$ and $\varphi (x,y) = (x,y, (12 -2x-3y )/6)$. Hence our integral is:
$$ \int_0^6\int_0^{(12 -2x)/3} \left( 3x\frac{12-2x-3y}{6}, y + \frac{12-2x-3y}{6}x, (xy+1)\frac{12-2x-3y}{6} \right) \begin{pmatrix} 1/3 \\ 1/2 \\ 1 \end{pmatrix}dydx \ . $$
That is,
$$ \int_0^6\int_0^{(12 -2x)/3} \left(\frac{y}{2} + \frac{12 -2x -3y}{4}x + \frac{12-2x-3y}{6} \right)dydx \ . $$
If $S_2$ is the face $y=0$ of $V$, then
$$ \int_{S_2}\langle F, dS \rangle = \int_{D_2}\left( 3xz, zx , z\right) \begin{pmatrix} 0 \\ -1 \\ 0 \end{pmatrix} dzdx = \int_0^6\int_0^{(12-x)/3}(-zx)dzdx \ . $$
And you should try to write down now $\int_{S_3}$ and $\int_{S_4}$ -if you haven't done yet.
On the other hand,
$$ \int_V \mathrm{div}F dxdydz = \int_0^6\int_0^{(12-2x)/3}\int_0^{(12-2x-3y)/6} (3z + 2 + xy)dzdydx \ . $$
So, if you understand those previous integrals and are able to write down those correspondint to the triangles $S_3$ and $S_4$, this would be enough. The rest of the computations are quite long and boring.