We can make the flux calculation for each surface directly by evaluating the surface integral $ \ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ $ , and also by applying the Divergence Theorem as a check.
For (1), the "upper" hemisphere of radius 2 centered on the origin has the equation $ \ z \ = \ \sqrt{4 \ - \ x^2 \ - \ y^2} \ $ . We may also write $ \ x^2 \ + \ y^2 \ + \ z^2 \ = \ 4 \ \ , \ z \ \ge \ 0 \ $ to determine the unit "outward" normal to the hemispherical surface, then continuing on to compute the flux integral:
a) using the gradient --
$$ g(x,y,z) \ = \ x^2 \ + \ y^2 \ + \ z^2 \ - \ 4 \ \ \Rightarrow \ \ \nabla g \ = \ \langle \ 2x \ , \ 2y \ , \ 2z \ \rangle $$
$$ \Rightarrow \ \ \| \nabla g \ \| \ = \ \sqrt{(2x)^2 \ + \ (2y)^2 \ + \ (2z)^2} \ = \ 2 \cdot 2 \ = \ 4 $$
$$ \Rightarrow \ \ \mathbf{\hat{n}} \ = \ \frac{\nabla g }{ \| \nabla g \ \|} \ = \ \langle \ \frac{x}{2} \ , \ \frac{y}{2} \ , \ \frac{z}{2} \ \rangle \quad \text{[unit normal]} $$
$$ \Rightarrow \ \ \mathbf{F} \cdot \mathbf{\hat{n}} \ = \ \langle \ x^2 \ , \ y^2 \ , \ z^2 \ \rangle \cdot \ \langle \ \frac{x}{2} \ , \ \frac{y}{2} \ , \ \frac{z}{2} \ \rangle \ = \ \frac{x^3 \ + \ y^3 \ + \ z^3}{2} \ \ . $$
This is well-suited to the use of spherical coordinates, so integrating over the hemispherical surface of radius $ \ R \ = \ 2 \ $ gives
$$ \ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS $$
$$ = \ \ \int_0^{2 \pi} \int_0^{\pi / 2} \ \frac{1}{2} ( \ R^3 \ \sin^3 \phi \ \cos^3 \theta \ + \ \ R^3 \ \sin^3 \phi \ \sin^3 \theta \ + \ R^3 \ \cos^3 \phi \ ) \ \ R^2 \ \sin \phi \ d\phi \ d\theta $$
$$ = \ \ \frac{1}{2}R^5 \ \int_0^{2 \pi} \int_0^{\pi / 2} \ ( \ \sin^4 \phi \ [ \ \cos^3 \theta \ + \ \sin^3 \theta \ ] \ + \ \cos^3 \phi \ \sin \ \phi \ ) \ \ d\phi \ d\theta $$
$$ = \ \ \frac{1}{2} \cdot \ 2^5 \ \left[ \ \int_0^{2 \pi} \int_0^{\pi / 2} \ \sin^4 \phi \ ( \ \cos^3 \theta \ + \ \sin^3 \theta \ ) \ \ d\phi \ d\theta \quad + \ \ \int_0^{2 \pi} \int_0^{\pi / 2} \ \cos^3 \phi \ \sin \ \phi \ \ d\phi \ d\theta \ \right] $$
$$ = \ \ 16 \ \left[ \ 0 \ + \ \int_0^{2 \pi} d\theta \ \int_0^{\pi / 2} \cos^3 \phi \ \sin \ \phi \ \ d\phi \ \right] \ = \ 16 \ \cdot \ 2 \pi \ \cdot \ \left( -\frac{1}{4} \cos^4 \phi \ \right) \vert_0^{\pi / 2} $$
[the first integral is zero since odd powers of cosine are integrated over one full period]
$$ = \ 16 \ \cdot \ 2 \pi \ \cdot \ \frac{1}{4} \ = \ 8 \pi \ \ . $$
b) using the graph function --
$$ z \ = \ g(x,y) \ = \ \sqrt{4 \ - \ x^2 \ - \ y^2} \ \ \Rightarrow \ \ z^2 \ = \ 4 \ - \ x^2 \ - \ y^2 $$
$$ \Rightarrow \ \ 2 z \ \frac{\partial g}{\partial x} \ = \ - 2 x \ \ , \ \ 2 z \ \frac{\partial g}{\partial y} \ = \ - 2 y $$
[using implicit differentiation]
$$ \Rightarrow \ \ \frac{\partial g}{\partial x} \ = \ - \frac{x}{z} \ \ , \ \ \frac{\partial g}{\partial y} \ = \ - \frac{y}{z} $$
$$ \Rightarrow \ \ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ \ = \ \ \iint_D \ -F_x \ \frac{\partial g}{\partial x} \ -F_y \ \frac{\partial g}{\partial y} \ + \ F_z \ \ dA $$
$$ = \ \ \iint_D \ -x^2 \left(- \frac{x}{z}\right) \ -y^2 \left(- \frac{y}{z}\right) \ + \ z^2 \ \ dA \ \ = \ \ \iint_D \ \left(\frac{x^3 \ + \ y^3}{z}\right) \ + \ z^2 \ \ dA \ \ . $$
This surface integral is performed over the projected area of the hemispherical surface onto the $ \ xy-$ plane, which is a disk of radius 2 ; this lends itself well to the use of polar coordinates:
$$ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS $$
$$ = \ \ \int_0^{2 \pi} \int_0^2 \ \frac{r^3 \ ( \ \cos^3 \theta \ + \ \sin^3 \theta \ )}{\sqrt{4 \ - \ r^2}} \ \ r \ dr \ d\theta \ \ + \ \ \int_0^{2 \pi} \int_0^2 \ ( \ 4 \ - \ r^2 ) \ \ r \ dr \ d\theta $$
$$ = \ \ 0 \ \ + \ \ \int_0^{2 \pi} d\theta \ \int_0^2 \ ( \ 4r \ - \ r^3 ) \ \ dr \ \ = \ 2 \pi \ \left( \ 2r^2 \ - \ \frac{1}{4}r^4 \right) \vert_0^2 $$
[the first integral again being zero because odd powers of cosine are integrated over one period]
$$ = \ 2 \pi \ ( \ 8 \ - \ 4 \ ) \ = \ 8 \pi \ \ . $$
This then is the "outward" flux through the hemispherical surface. We can apply the Divergence Theorem over the volume of the hemisphere as a check:
$$ \ \nabla \cdot \mathbf{F} \ = \ 2x \ + \ 2y \ + 2z $$
$$ \Rightarrow \ \ \iiint_V \ \nabla \cdot \mathbf{F} \ \ dV \ \ = \ \ \iiint_V \ ( \ 2x \ + \ 2y \ + 2z \ ) \ \ dV $$
$$ = \ \ 2 \ \int_0^{2 \pi} \int_0^{\pi / 2} \int_0^2 \ r \ ( \ \sin \phi \ \cos \theta \ + \ \sin \phi \ \sin \theta \ + \ \cos \phi \ ) \ \ r^2 \ dr \ \sin \phi \ d\phi \ d\theta $$
$$ = \ \ 2 \ \int_0^{2 \pi} \int_0^{\pi / 2} \int_0^2 \ r^3 \ ( \ \sin^2 \phi \ [ \ \cos \theta \ + \ \sin \theta \ ] \ + \ \cos \phi \ \sin \phi \ ) \ \ dr \ d\phi \ d\theta $$
$$ = \ \ 2 \ \left[ \ \int_0^{2 \pi} \int_0^{\pi / 2} \int_0^2 \ r^3 \ \sin^2 \phi \ [ \ \cos \theta \ + \ \sin \theta \ ] \ \ dr \ d\phi \ d\theta \quad + \ \ \int_0^{2 \pi} \int_0^{\pi / 2} \int_0^2 \ r^3 \ \cos \phi \ \sin \phi \ \ dr \ d\phi \ d\theta \ \right] $$
$$ = \ \ 2 \ \left[ \ 0 \ \ + \ \ \int_0^{2 \pi} d\theta \ \int_0^{\pi / 2} \cos \phi \ \sin \phi \ \ d\phi \ \int_0^2 \ r^3 \ \ dr \ \right] $$
[the first integral gives zero since sine and cosine functions are integrated over one period]
$$ = \ \ 2 \ \cdot \ 2 \pi \ \int_0^{\pi / 2} \ \frac{1}{2} \sin \ 2 \phi \ \ d\phi \ \int_0^2 \ r^3 \ \ dr \ \ = \ 2 \ \cdot \ 2 \pi \ \left( \ -\frac{1}{4} \cos \ 2 \phi \ \right) \vert_0^{\pi / 2} \ \cdot \ \left( \ \frac{1}{4}r^4 \ \right) \vert_0^2 $$
$$ = \ \ 2 \ \cdot \ 2 \pi \ \left( -\frac{1}{4} \right) \ ( \ [-1] \ - \ 1 \ ) \ \cdot \ \frac{1}{4} \ \cdot \ 2^4 \ = \ 8 \pi \ \ . $$
The outward flux through the hemispherical surface is equal to this amount less the flux through the base of the hemisphere. But this circular base (of radius 2) lies in the $ \ xy-$ plane ( $ \ z \ = \ 0 \ $ ) , so we have
$$ \iint_B \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ \ = \ \ \iint_B \ \langle \ x^2 \ , \ y^2 \ , \ 0^2 \ \rangle \cdot \langle \ 0 \ , \ 0 \ , \ -1 \ \rangle \ \ dS \ = \ 0 \ \ . $$
As there is no $ \ z-$ component of the field $ \ \mathbf{F} \ $ in the $ \ xy-$ plane, there is no flux through the base of the hemisphere. Hence, we confirm our result for the flux through the hemispherical surface.
$$ \ \ $$
For (2), we deal with the "upper" nappe of the cone having the equation $ \ z \ = \ 2 \ \sqrt{x^2 \ + \ y^2} \ $ , or $ \ z^2 \ = \ 4x^2 \ + \ 4y^2 \ \ , \ z \ \ge \ 0 \ $ . Many of the aspects of this problem are similar to the first one, so we will elaborate less on the details. One peculiarity here is the the "upward" normal to the conical surface is chosen, which points "into" the volume of the nappe; thus, we take the negative of the standard definition for the normal vector.
a) using the gradient --
$$ g(x,y,z) \ = \ 4x^2 \ + \ 4y^2 \ - \ z^2 \ \ \Rightarrow \ \ \nabla g \ = \ \langle \ 8x \ , \ 8y \ , \ -2z \ \rangle $$
$$ \Rightarrow \ \ \| \nabla g \ \| \ = \ \sqrt{(8x)^2 \ + \ (8y)^2 \ + \ (-2z)^2} \ = \ \sqrt{64 \ (x^2 \ + \ y^2) \ + \ 4z^2} $$
$$ = \ \sqrt{16z^2 \ + \ 4z^2} \ = \ 2 \ \sqrt{5} \ z $$
$$ \Rightarrow \ \ \mathbf{\hat{n}} \ = \ -\frac{\nabla g }{ \| \nabla g \ \|} \ = \ -\frac{1}{\sqrt{5}} \ \langle \ \frac{4x}{z} \ , \ \frac{4y}{z} \ , \ -1 \ \rangle \quad \text{[ "upward" unit normal]} $$
$$ \Rightarrow \ \ \mathbf{F} \cdot \mathbf{\hat{n}} \ = \ \frac{1}{\sqrt{5}} \left( -\frac{4x^3}{z} \ - \ \frac{4y^3}{z} \ + \ z^2 \right) $$
$$ \Rightarrow \ \ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ \ = \ \ \frac{1}{\sqrt{5}} \ \int_0^{2 \pi} \int_0^1 \ \left[ \ - \frac{4r^3 \ (\cos^3 \theta \ + \ \sin^3 \theta) }{2r} \ + \ (2r)^2 \ \right] \ \ (\sqrt{5}) \ r \ dr \ d\theta $$
[here, we use cylindrical coordinates: we thus have $ \ z \ = \ 2r \ $ and the factor of $ \ \sqrt{5} \ $ is introduced in the integration on the conical surface, since the slope of the cone "wall" is 2]
$$ = \ \ \int_0^{2 \pi} d\theta \ \int_0^1 \ 4r^3 \ \ \ dr \ \ = \ 2 \pi \ ( \ r^4 \ ) \vert_0^1 \ = \ 2 \pi \ \ . $$
[the integration in theta yielding zero]
b) using the graph function --
$$ z \ = \ g(x,y) \ = \ 2 \ \sqrt{x^2 \ + \ y^2} \ \ \Rightarrow \ \ z^2 \ = \ 4 x^2 \ + \ 4 y^2 $$
$$ \Rightarrow \ \ 2 z \ \frac{\partial g}{\partial x} \ = \ 8 x \ \ , \ \ 2 z \ \frac{\partial g}{\partial y} \ = \ 8 y \ \ \Rightarrow \ \ \frac{\partial g}{\partial x} \ = \ \frac{4x}{z} \ \ , \ \ \frac{\partial g}{\partial y} \ = \ \frac{4y}{z} $$
$$ \Rightarrow \ \ \iint_S \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ \
= \ \ \iint_D \ -x^2 \left( \frac{4x}{z}\right) \ -y^2 \left( \frac{4y}{z}\right) \ + \ z^2 \ \ dA $$
$$ = \ \ \iint_D \ -\left(\frac{4x^3 \ + \ 4y^3}{z}\right) \ + \ z^2 \ \ dA \ \ , $$
giving us the same integration over the disk of radius 1 on the $ \ xy-$ plane, which is the projection of the conical surface, and therefore the same result for the "upward" flux through the conical surface.
Making a check using the Divergence Theorem, we integrate in cylindrical coordinates over the volume of the cone up to $ \ z \ = \ 2 \ $ to obtain
$$ \iiint_V \ \nabla \cdot \mathbf{F} \ \ dV \ \ = \ \ \iiint_V \ ( \ 2x \ + \ 2y \ + 2z \ ) \ \ dV $$
$$ = \ \ 2 \ \int_0^{2 \pi} \int_0^1 \int_0^{2r} \ ( \ r \ \cos \theta \ + \ r \ \sin \theta \ + \ z \ ) \ \ dz \ r \ dr \ d\theta $$
$$ = \ \ 2 \ \int_0^{2 \pi} \int_0^1 \ \left( \ rz \ [ \ \cos \theta \ + \ \sin \theta \ ] \ + \ \frac{1}{2}z^2 \ \right) \vert_0^{2r} \ \ r \ dr \ d\theta $$
$$ = \ \ 2 \ \int_0^{2 \pi} \int_0^1 \ 2r^3 \ ( \ \cos \theta \ + \ \sin \theta \ ) \ + \ \frac{1}{2} \ (2r)^2 \cdot r \ \ \ dr \ d\theta $$
$$ = \ \ 2 \ \int_0^{2 \pi} d\theta \ \int_0^1 \ 2r^3 \ \ \ dr \ \ = \ 2 \ \cdot \ 2 \pi \ \left(\frac{1}{2}r^4 \right) \vert_0^1 \ = \ 2 \ \cdot \ 2 \pi \ \frac{1}{2} \ = \ 2 \pi \ \ . $$
At the level $ \ z \ = \ 2 \ $ , the field is $ \ \mathbf{F} \ = \ \langle \ x^2 \ , \ y^2 \ , \ 2^2 \ \rangle \ $ , hence for the top surface of the conical volume,
$$ \mathbf{F} \cdot \mathbf{\hat{n}} \ = \ \langle \ x^2 \ , \ y^2 \ , \ 4 \ \rangle \cdot \ \langle \ 0 \ , \ 0 \ , \ 1 \ \rangle \ = \ 4 \ \ . $$
The "upward" flux through this surface, a circle of radius 1, is then
$$ \iint_T \ \mathbf{F} \cdot \mathbf{\hat{n}} \ \ dS \ \ = \ 4 \ \cdot \ \pi \ \cdot \ 1^2 \ = \ 4 \pi \ \ . $$
With this amount of "upward" flux through the top of the conical volume and a net "upward" flux of $ \ 2 \pi \ $ through that volume, the "upward" flux through the cone "wall" must be $ \ 2 \pi \ $ , as we have found from the flux integration.
$$ \ \ $$
One additional comment may be made here. Through all of this discussion, we have found the portion of the flux integrations involving the azimuthal angle $ \ \theta \ $ to always be zero. The geometrical interpretation for this is that the components of the field $ \ \mathbf{F} \ $ that are parallel to the $ \ xy-$ plane are not only non-negative, but are symmetrical about the line $ \ y = \ x \ $ . Because the "horizontal" cross-sections of both the hemisphere and the cone are circular and centered on the $ \ z-$ axis, the flux entering the boundaries of these cross-sections on one side of the line $ \ y = -x \ $ exactly matches the flux leaving these boundaries on the other side of the line (as may be seen in the graph below). So only the $ \ z-$ component of the field makes any contribution to the net flux through the surfaces we have examined.
Best Answer
Using Stokes' theorem the integral is
$$I =\int_S \text{curl } \mathbf{F} \cdot \mathbf{n} \,dS = \oint_C \mathbf{F} \cdot d \,\mathbf{l} $$
where $C$ is the circle of radius $r$ centered at the origin of the $xy-$plane.
On $C$ we have, using polar coordinates, $F_x = xy^2 \tanh(x^2) = r^3\cos \theta\sin^2 \theta \tanh(r^2 \cos^2 \theta)$ and $F_y = x = r \cos \theta$. The unit tangent to $C$ is $- \sin \theta \, \mathbf{e}_x + \cos \theta \, \mathbf{e}_y$, with no $z-$component. Consequently,
$$I = \int_0^{2\pi} r^3\cos\theta \sin^2 \theta \tanh(r^2 \cos^2 \theta) (-\sin \theta) r \, d \theta + \int_0^{2 \pi}r \cos \theta (\cos \theta) r\,d \theta \\ = -r^4 \underbrace{\int_0^{2\pi}\cos \theta \sin^3 \theta \tanh(r^2\cos^2 \theta) \, d \theta}_{= \,0 \text{ by symmetry}} \, + \,r^2\int_0^{2\pi}\cos^2 \theta \, d \theta \\ = \pi r^2$$