A good parametrization for your surface is:
$$\mathbf r(\theta,\phi)=\langle x= 2\sin\phi\cos\theta,y=2\sin\phi\sin\theta,z=2\cos\phi\rangle.$$
Where $\phi$ is the angle between the positive $z$-axis and the vector $\boldsymbol v$ with tail on the origin and tip at a point on the sphere, and $\theta$ is the angle between the positive $x$-axis and the projection of the vector $\boldsymbol v$ onto the $xy$-plane. So your limits of integration would be:
$$0\leq\theta\leq 2\pi,\\0\leq\phi\leq\pi,\\0\leq\rho\leq2.$$
Divergence theorem tells you that:
$$\iint\limits_S \mathbf F \cdot d\mathbf S = \iiint\limits_E \text{div}\mathbf F\,dV.$$
The last triple integral by Fubini is the iterated integral with the bounds I proposed, do change of variables and don't forget the jacobian $\rho^2\sin\phi$.
Your problem is the formula for the flux. To calculate the flux, you don't need the curl, just use the surface integral
$$ \int\int_D F\cdot n\ dA $$
where $n$ is the normal vector to the surface $D$.
When $D$ is $S_1$, the integral becomes
$$ \int_0^1 \int_0^{1-x} zy \ dy\ dx $$
This is because $S_1$ lies in the $xy$-plane, so $n = (0,0,1)$. So $F\cdot n = zy$. Since $z=0$ on $S_1$, this is just $0$.
Similarly: When $D$ is $S_2$, the corresponding integral is
$$ \int_0^1 \int_0^{1-x} y^2\ dz \ dx .$$
Snince $y=0$ on $S_2$, the integral has value $0$.
When $D$ is $S_3$, the corresponding integral is
$$ \int_0^1 \int_0^{1-y} xy\ dz\ dy. $$
Since $x=0$ on $S_3$, the integral has value $0$.
When $D$ is $S_4$, the corresponding integral is
$$ \int_0^1 \int_0^{1-x} xy + y^2 + (1-x-y)y\ dy \ dx .$$
Here, the normal vector is $(1,1,1)$. So the dot product $F \cdot n = xy + y^2 + zy$. Notice that on $S_4$, we can substitute $z = 1-z-y$. The value of this integral is $\frac{1}{6}$.
Thus, the total flux of the pyramid is $0+0+0+\frac{1}{6} = \boxed{\frac{1}{6}}$.
As for part (b), the correct integral limits should be
$$ \int_0^1 \int_0^{1-x} \int_0^{1-x-y} 4y\ dz\ dy\ dx$$
and this evaluates to $\boxed{\frac{1}{6}}$.
Best Answer
We have $x=r\cos\theta+1$ and $y=r\sin\theta$. Hence, $x-y=r(\cos\theta-\sin\theta)+1.$ This shouldn't be too difficult to integrate.