I know I've got a mistake, probably with question a. But I don't know where or what the mistake is.
The question is
Let $D\subset\mathbb{R}^3$ be the pyramid with vertices $(0,0,0)$, $(1,0,0)$, $(0,1,0)$, $(0,0,1)$.
Given is the vector field $\vec{F} = xy\hat{i}+y^2\hat{j}+zy\hat{k}$.
Calculate the flux of $\vec{F}$ through $D$.
a) Directly.
b) Using divergence theorem.
My solution of a:
Since we have a pyramid of four surfaces, we shall calculate the flux through each separately. Let $S_1$ denote the surface on the $xy$-plane. $S_2$ on the $xz$-plane. $S_3$ on the $zy$-plane. And $S_4$ in the $xyz$-space.
Let the vector field be in the form $\vec{F}=P\hat{i}+Q\hat{j}+R\hat{k}$, then I calculate the flux by $$\int\int_D \left(-P\cdot g_x(x,y) -Q\cdot g_y(x,y) + R\right)\,dA$$
The flux of $\vec{F}$ through $S_1$ is given by $$\int_{x=0}^{1}\int_{y=0}^{1-x}\left((-xy)\cdot g_x-y^2\cdot g_y + zy\right)\,dydx =\int_{x=0}^{1}\int_{y=0}^{1-x}\left(xy+y^2 + y -xy -y^2\right)\,dydx$$, with $g(x,y)=1-x-y$ for $x+y=1$. If I calculate this integral I get $1/6$.
The flux of $\vec{F}$ through $S_2$ is given by $$\int_{x=0}^{1}\int_{z=0}^{1-x}\left((-xy)\cdot g_x+y^2 + zy\cdot g_z\right)\,dzdx$$, with $g(x,z)=1-x-z$ for $x+z=1$. If I calculate this integral I get $1/6$.
The flux of $\vec{F}$ through $S_3$ is given by $$\int_{y=0}^{1}\int_{z=0}^{1-y}\left((xy+y^2\cdot g_y + zy\cdot g_z\right)\,dzdx$$, with $g(y,z)=1-y-z$ for $y+z=1$. If I calculate this integral I get $1/6$.
The flux of $\vec{F}$ through $S_4$ is given by $$\int_{x=0}^{1}\int_{y=0}^{1}\left((-xy)\cdot g_x-y^2\cdot g_y + zy\right)\,dydx $$, with $g(x,y)=1-x-y$. If I calculate this integral I get $1/2$.
The total flux is then the flux through all the separate surfaces added together, we get a flux of $1$.
My solution of b:
Since the surface is simple and closed, we get
$$\int_{x=0}^1\int_{y=0}^1\int_{z=0}^1 Div(\vec{F})dz\,dy\,dx$$ with the divergence equal to $y+2y+y=4y$. Hence the flux is $2$.
Best Answer
Your problem is the formula for the flux. To calculate the flux, you don't need the curl, just use the surface integral
$$ \int\int_D F\cdot n\ dA $$
where $n$ is the normal vector to the surface $D$.
When $D$ is $S_1$, the integral becomes $$ \int_0^1 \int_0^{1-x} zy \ dy\ dx $$
This is because $S_1$ lies in the $xy$-plane, so $n = (0,0,1)$. So $F\cdot n = zy$. Since $z=0$ on $S_1$, this is just $0$.
Similarly: When $D$ is $S_2$, the corresponding integral is
$$ \int_0^1 \int_0^{1-x} y^2\ dz \ dx .$$
Snince $y=0$ on $S_2$, the integral has value $0$.
When $D$ is $S_3$, the corresponding integral is
$$ \int_0^1 \int_0^{1-y} xy\ dz\ dy. $$
Since $x=0$ on $S_3$, the integral has value $0$.
When $D$ is $S_4$, the corresponding integral is
$$ \int_0^1 \int_0^{1-x} xy + y^2 + (1-x-y)y\ dy \ dx .$$
Here, the normal vector is $(1,1,1)$. So the dot product $F \cdot n = xy + y^2 + zy$. Notice that on $S_4$, we can substitute $z = 1-z-y$. The value of this integral is $\frac{1}{6}$.
Thus, the total flux of the pyramid is $0+0+0+\frac{1}{6} = \boxed{\frac{1}{6}}$.
As for part (b), the correct integral limits should be
$$ \int_0^1 \int_0^{1-x} \int_0^{1-x-y} 4y\ dz\ dy\ dx$$
and this evaluates to $\boxed{\frac{1}{6}}$.