The result of the line integral is correct, so the mistake must be with your calculation of the surface integral. You have $\nabla \times \vec F = \left( 0,1-2xz,y^2 \right)$ and with a surface of the form $z=g(x,y)$, the normal $\vec n$ is given by $\left(-g_x,-g_y,1\right)$. The projection of the surface onto the $xy$-plane is a disc centered in the origin and with radius $\sqrt{6}$, so you have to integrate over this disc $D$:
$$\iint_D \left( 0,1-2xz,y^2 \right) \cdot \left(-g_x,-g_y,1\right) \,\mbox{d}A$$
where $z=g(x,y)$ with $g(x,y)=x^2+y^2$, so:
$$\iint_D \left( 0,1-2x\left(x^2+y^2\right),y^2 \right) \cdot \left(-2x,-2y,1\right) \,\mbox{d}A =
\iint_D \left( 4 x^3 y + 4 x y^3 + y^2 - 2 y \right)
\,\mbox{d}x\,\mbox{d}y$$
You can choose to switch to polar coordinates or not, but you should find $9\pi$ either way.
Maybe this is sufficient to find your mistake? If not, perhaps you can show us your calculations.
Method 1: Calculating the Line Integral.
Lookings at the plots of the surfaces
$$ \color{blue}{z = y^2} \qquad \color{orange}{x^2+y^2=1} $$
We see that the intersecting curve $C$ looks like:
We can parametrize this curve by seeing that, since the $x,y$ coordinates lie on the unit circle, we have:
$$C: \mathbf{x}(t) = \langle \cos t, \sin t, \sin^2 t \rangle \qquad (0 \leq t \leq 2\pi) $$
Taking the vector field:
$$ \mathbf{F}(x,y,z) = \langle 2yz, xz, xy \rangle $$
We evaluate the line integral:
\begin{align*}
\int_C \mathbf{F} \cdot d\mathbf{s} &= \int_0^{2\pi} \mathbf{F}(\mathbf{x}(t)) \cdot \mathbf{x}'(t) \, dt \\
&= \int_0^{2\pi} \langle 2\sin^3 t, \cos t \sin^2 t, \cos t \sin t \rangle \cdot \langle -\sin t, \cos t, 2\sin t \cos t \rangle \, dt \\
&= \int_0^{2\pi} \left(3 \cos^2 t \sin^2 t - 2\sin^4 t \right) \, dt \\
&= \int_0^{2\pi} \left[3\left(\frac{\sin 2t}{2} \right)^2 - 2\left(\frac{1 - \cos 2t}{2} \right)^2 \right] \, dt \\
&= \int_0^{2\pi} \left[\frac{3\sin^2 2t}{4} - \frac{2 - 4\cos 2t + 2\cos^2 2t}{4} \right] \, dt \\
&= \int_0^{2\pi} \left[\frac{3\sin^2 2t - 2\cos^2 t + 4\cos 2t - 2}{4} \right] \, dt \\
&= \int_0^{2\pi} \left[\frac{\sin^2 2t + 3\cos 2t - 2}{4} \right] \, dt \\
&= \int_0^{2\pi} \left[\frac{1 - \cos 4t + 6\cos 2t - 4}{8} \right] \, dt \\
&= \int_0^{2\pi} \left[\frac{1 - 4}{8} \right] \, dt \\
&= -\frac{3}{8}\int_0^{2\pi} \, dt \\
&= -\frac{3\pi}{4}
\end{align*}
Method 2: Applying Stokes' Theorem.
We must choose a surface $S$ that has $C$ as its boundary. We can simply choose the part of the surface $\color{blue}{z = y^2}$ thats enclosed within $\color{orange}{x^2+y^2=1}$, that is:
$$S = \{ (x,y,z) \mid x^2 + y^2 \leq 1 \,,\, z = y^2 \}$$
We can parametrize the surface by noting that the $(x,y)$ points form a solid disk in the $x$-$y$ space, and $z$ is simply $y^2$, so we parametrize the surface as
$$S: \quad \mathbf{X}(r,\theta) = \langle r \cos \theta, r \sin \theta, r^2 \sin^2 \theta \rangle \qquad (\underbrace{0 \leq r \leq 1, 0 \leq \theta \leq 2\pi}_{(r,\theta) \in \, R = [0,1] \times [0,2\pi]})$$
We find the normal vector that arises from this parametrization by calculating:
\begin{align*}
\frac{\partial \mathbf{X}}{\partial r} &= \langle \cos \theta, \sin \theta, 2r \sin^2 \theta \rangle \\
\frac{\partial \mathbf{X}}{\partial \theta} &= \langle -r \sin \theta, r\cos \theta, 2r^2 \sin \theta \cos \theta \rangle \\
\mathbf{N}(r,\theta) &= \frac{\partial \mathbf{X}}{\partial r} \times \frac{\partial \mathbf{X}}{\partial \theta} \\
&= \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\
\cos \theta & \sin \theta & 2r \sin^2 \theta \\
-r \sin \theta & r\cos \theta & 2r^2 \sin \theta \cos \theta
\end{vmatrix} \\
&= \langle 0, -2r^2 \sin \theta, r \rangle
\end{align*}
We note that the $z$ component of the normal vector is positive, so as to point upward, conforming with the counterclockwise orientation of the line integral taken before.
The curl of the vector field $\mathbf{F}$ is:
\begin{align*}
\nabla \times \mathbf{F}
&= \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\
\frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\
2yz & xz & xy
\end{vmatrix}
= \langle 0, y, -z \rangle
\end{align*}
And so, we evaluate the line integral over $C = \partial S$ by applying Stoke's Theorem:
\begin{align*}
\int_{\partial S} \mathbf{F} \cdot d\mathbf{s} &= \iint_S (\nabla \times \mathbf{F}) \cdot d\mathbf{S} \\
&= \iint_R (\nabla \times \mathbf{F})\mid_{\mathbf{X}(r,\theta)} \cdot \mathbf{N}(r,\theta) \, dA \\
&= \int_0^{2\pi} \int_0^1 \langle 0, r \sin \theta, -r^2 \sin^2 \theta \rangle \cdot \langle 0, -2r^2 \sin \theta, r \rangle \, dr \, d\theta \\
&= \int_0^{2\pi} \int_0^1 (-3r^3 \sin^2 \theta) \, dr \, d\theta \\
&= -\frac{3}{4}\int_0^{2\pi} \sin^2 \theta \, d\theta \\
&= -\frac{3}{8}\int_0^{2\pi} (1 - \cos 2\theta) \, d\theta \\
&= -\frac{3}{8}\int_0^{2\pi} 1 \, d\theta \\
&= -\frac{3\pi}{4}
\end{align*}
Best Answer
For the first integral, you forgot to take the derivative of $r$ inside the integral. If you fix that you will quickly find $\pi$ as expected.
For the second integral, your expression of $\vec{n}$ is correct but then you go wrong after that. You need to parametrize the hemisphere (e.g. with spherical coordinates) and then find the correct expression of $\vec{dS} = \vec{n}dS$ and $\vec{F}$ in these coordinates. You can try spherical coordinates $x = \sin \theta \cos \phi$, $y = \sin \theta \sin \phi$, $z = \cos \theta$ where $\theta \in [0, \pi/2]$ and $\phi \in [0,2\pi]$ (colatitude and azimuth). In this case $dS = (\sin \theta) d\theta d\phi$, and it is actually quick to see that the integral $\iint \mathrm{curl}(\vec{F}) \cdot \vec{dS} = \int_{\theta = 0}^{\pi/2} \int_{\phi=0}^{2\pi} \vec{F} \cdot \vec{n}\, dS$ is equal to $\pi$.