The result of the line integral is correct, so the mistake must be with your calculation of the surface integral. You have $\nabla \times \vec F = \left( 0,1-2xz,y^2 \right)$ and with a surface of the form $z=g(x,y)$, the normal $\vec n$ is given by $\left(-g_x,-g_y,1\right)$. The projection of the surface onto the $xy$-plane is a disc centered in the origin and with radius $\sqrt{6}$, so you have to integrate over this disc $D$:
$$\iint_D \left( 0,1-2xz,y^2 \right) \cdot \left(-g_x,-g_y,1\right) \,\mbox{d}A$$
where $z=g(x,y)$ with $g(x,y)=x^2+y^2$, so:
$$\iint_D \left( 0,1-2x\left(x^2+y^2\right),y^2 \right) \cdot \left(-2x,-2y,1\right) \,\mbox{d}A =
\iint_D \left( 4 x^3 y + 4 x y^3 + y^2 - 2 y \right)
\,\mbox{d}x\,\mbox{d}y$$
You can choose to switch to polar coordinates or not, but you should find $9\pi$ either way.
Maybe this is sufficient to find your mistake? If not, perhaps you can show us your calculations.
The paraboloid $z=x^2+y^2$ and the plane $z=ax+by$ intersect for
$$x^2+y^2=ax+by\implies\left(x-\frac a2\right)^2+\left(y-\frac b2\right)^2=\frac{a^2+b^2}4$$
i.e. in the cylinder with cross sections parallel to the $x$-$y$ plane centered at $\left(\frac a2,\frac b2\right)$ with radius $\frac{\sqrt{a^2+b^2}}2$. Naturally, I think this suggests a parametrization of $S$ in cylindrical coordinates,
$$\hat s(u,v)=x(u,v)\,\hat\imath+y(u,v)\,\hat\jmath+z(u,v)\,\hat k$$
where
$$\begin{cases}x(u,v)=u\cos v+\frac a2\\[1ex]y(u,v)=u\sin v+\frac b2\\[1ex]z(u,v)=x(u,v)^2+y(u,v)^2=\frac{a^2+b^2}4+u^2+au\cos v+bu\sin v\end{cases}$$
with $0\le u\le\frac{\sqrt{a^2+b^2}}2$ and $0\le v\le2\pi$.
From the integrand we can obtain the underlying vector field $\hat F$:
$$y^2\,\mathrm dz=\underbrace{(0\,\hat\imath+0\,\hat\jmath+y^2\,\hat k)}_{\hat F(x,y,z)}\cdot(\mathrm dx\,\hat\imath+\mathrm dy\,\hat\jmath+\mathrm dz\,\hat k)$$
Compute the curl:
$$\nabla\times\hat F=2y\,\hat\imath$$
Take the normal vector to $S$ to be $\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}$:
$$\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}=(au+2u^2\cos v)\,\hat\imath+(bu+2u^2\sin v)\,\hat\jmath-u\,\hat k$$
Then by Stokes' theorem the integral is
$$\begin{align*}
\oint_Cy^2\,\mathrm dz&=\oint_C\hat F\cdot\mathrm d\hat r\\[1ex]
&=\iint_S(\nabla\times\hat F)\cdot\mathrm d\hat S\\[1ex]
&=\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}2}\left(2y(u,v)\,\hat\imath\right)\cdot\hat n\,\mathrm du\,\mathrm dv\\[1ex]
&=2\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}/2}\left(u\sin v+\frac b2\right)\,\mathrm du\,\mathrm dv\\[1ex]
&=\boxed{2\pi b\sqrt{a^2+b^2}}
\end{align*}$$
Best Answer
Given vector field $$ \vec F = y^2 \hat i + x \hat j + z^2 \hat k$$ $$\Rightarrow \nabla \times \vec F = (1 - 2y) \hat k $$
Generally, $$ \hat n dS = \bigg( - \cfrac{ \partial z}{ \partial x} \hat i - \cfrac{ \partial z}{ \partial y} \hat j + \hat k \bigg) dA$$
Putting this all, $$ \iint_S ( \nabla \times \vec F) \cdot \hat n dS = \iint_A (1 - 2y) dA$$
On using polar coordinates, $dA = r dr d \theta, y = r \sin \theta$ and $ r \in [0,1] , \theta \in [0,2 \pi] $
The required integral becomes, $$\int^{2 \pi}_0 \int^1_0 (1 - 2 r \sin \theta ) r dr d\theta = \pi $$ QED