Given vector field $$ \vec F = y^2 \hat i + x \hat j + z^2 \hat k$$ $$\Rightarrow \nabla \times \vec F = (1 - 2y) \hat k $$
Generally, $$ \hat n dS = \bigg( - \cfrac{ \partial z}{ \partial x} \hat i - \cfrac{ \partial z}{ \partial y} \hat j + \hat k \bigg) dA$$
Putting this all, $$ \iint_S ( \nabla \times \vec F) \cdot \hat n dS = \iint_A (1 - 2y) dA$$
On using polar coordinates, $dA = r dr d \theta, y = r \sin \theta$ and $ r \in [0,1] , \theta \in [0,2 \pi] $
The required integral becomes, $$\int^{2 \pi}_0 \int^1_0 (1 - 2 r \sin \theta ) r dr d\theta = \pi $$
QED
The paraboloid $z=x^2+y^2$ and the plane $z=ax+by$ intersect for
$$x^2+y^2=ax+by\implies\left(x-\frac a2\right)^2+\left(y-\frac b2\right)^2=\frac{a^2+b^2}4$$
i.e. in the cylinder with cross sections parallel to the $x$-$y$ plane centered at $\left(\frac a2,\frac b2\right)$ with radius $\frac{\sqrt{a^2+b^2}}2$. Naturally, I think this suggests a parametrization of $S$ in cylindrical coordinates,
$$\hat s(u,v)=x(u,v)\,\hat\imath+y(u,v)\,\hat\jmath+z(u,v)\,\hat k$$
where
$$\begin{cases}x(u,v)=u\cos v+\frac a2\\[1ex]y(u,v)=u\sin v+\frac b2\\[1ex]z(u,v)=x(u,v)^2+y(u,v)^2=\frac{a^2+b^2}4+u^2+au\cos v+bu\sin v\end{cases}$$
with $0\le u\le\frac{\sqrt{a^2+b^2}}2$ and $0\le v\le2\pi$.
From the integrand we can obtain the underlying vector field $\hat F$:
$$y^2\,\mathrm dz=\underbrace{(0\,\hat\imath+0\,\hat\jmath+y^2\,\hat k)}_{\hat F(x,y,z)}\cdot(\mathrm dx\,\hat\imath+\mathrm dy\,\hat\jmath+\mathrm dz\,\hat k)$$
Compute the curl:
$$\nabla\times\hat F=2y\,\hat\imath$$
Take the normal vector to $S$ to be $\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}$:
$$\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}=(au+2u^2\cos v)\,\hat\imath+(bu+2u^2\sin v)\,\hat\jmath-u\,\hat k$$
Then by Stokes' theorem the integral is
$$\begin{align*}
\oint_Cy^2\,\mathrm dz&=\oint_C\hat F\cdot\mathrm d\hat r\\[1ex]
&=\iint_S(\nabla\times\hat F)\cdot\mathrm d\hat S\\[1ex]
&=\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}2}\left(2y(u,v)\,\hat\imath\right)\cdot\hat n\,\mathrm du\,\mathrm dv\\[1ex]
&=2\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}/2}\left(u\sin v+\frac b2\right)\,\mathrm du\,\mathrm dv\\[1ex]
&=\boxed{2\pi b\sqrt{a^2+b^2}}
\end{align*}$$
Best Answer
Parameterizing an upper hemisphere in spherical coordinates is no problem at all...
Stokes's Theorem states;
$$\int_{S}({\nabla \times F})\cdot dS=\oint_{\partial S}F\cdot ds$$
So, for our case we have;
$$\int_{S}[({\nabla \times \vec F})\cdot{\hat r}] dA=R\int_{0}^{2\pi}[\vec F\cdot {\hat\theta}] d\theta$$
$$\int_{S}[({\nabla \times \vec F})\cdot{\hat r}] dA=R\int_{0}^{2\pi}[R\sin(\theta)\cos({\pi\over2})] d\theta$$
$$\therefore \int_{S}[({\nabla \times \vec F})\cdot{\hat r}] dA=0$$
The formula for curl in spherical coordinates is a bit cumbersome but eventually you will arrive at...
$$\int_{S}[({\nabla \times \vec F})\cdot{\hat r}] dA$$
$$=R^2\left(\int_{0}^{\pi}(\cos^2(\phi)-\sin^2(\phi))d\phi\int_{0}^{2\pi}\sin(\theta)d\theta \space-\space \int_{0}^{\pi}\sin(\phi)d\phi\int_{0}^{2\pi}\cos(\theta)\sin(\theta)d\theta\right)$$
$$=0$$
So, as expected, we have;
$$\int_{S}({\nabla \times F})\cdot dS=\oint_{\partial S}F\cdot ds=0$$