In fact, the only constraints for the vector $\bf{n}$ are
$1.$ The vector $\bf{n}$ is a unit vector normal to the surface.
$2.$ It should have proper orientation depending on the orientation of the surrounding curve.
So, I think you may have made a mistake in the problem you solved and hence we may help you if you write it down in your question. :)
Verifying Stokes Theorem For Your Question
Your surface is enclosed by the intersection curve of the plane $x+z=1$ and the cylinder $x^2+y^2=36$ as the following figure shows.
The parametric equation of the intersection curve, the tangent vector, and the vector field are
$$\eqalign{
& {\bf{x}} = 6\cos \theta {\bf{i}} + 6\sin \theta {\bf{j}} + \left( {1 - 6\cos \theta } \right){\bf{k}} \cr
& {{d{\bf{x}}} \over {d\theta }} = - 6\sin \theta {\bf{i}} + 6\cos \theta {\bf{j}} + 6\sin \theta {\bf{k}} \cr
& F({\bf{x}}) = xy{\bf{i}} + 2z{\bf{j}} + 6y{\bf{k}} \cr} $$
and hence the line integral will be
$$\eqalign{
& I = \int\limits_C {F({\bf{x}}) \cdot {{d{\bf{x}}} \over {d\theta }}d\theta } = \int_{\theta = 0}^{2\pi } {\left( { - 6\sin \theta xy + 12\cos \theta z + 36\sin \theta y} \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta + 2\cos \theta \left( {1 - 6\cos \theta } \right) + 36{{\sin }^2}\theta } \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta - 12{{\cos }^2}\theta + 36{{\sin }^2}\theta + 2\cos \theta } \right)d\theta } \cr
& \,\,\, = 6\left[ { - 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta \cos \theta d\theta } - 12\int_{\theta = 0}^{2\pi } {{{\cos }^2}\theta d\theta } + 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta d\theta + 2\int_{\theta = 0}^{2\pi } {\cos \theta d\theta } } } \right] \cr
& \,\,\, = 6\left[ { - 36\left( 0 \right) - 12\left( \pi \right) + 36\left( \pi \right) + 2\left( 0 \right)} \right] \cr
& \,\,\, = 144\pi \cr} $$
Next, compute the area element vector $d\bf{S}$ and $\nabla \times {\bf{F}}$
$$\eqalign{
& {\bf{x}} = x{\bf{i}} + y{\bf{j}} + \left( {1 - x} \right){\bf{k}} \cr
& d{\bf{S}} = \left( {{{\partial {\bf{x}}} \over {\partial x}} \times {{\partial {\bf{x}}} \over {\partial y}}} \right)dxdy = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
1 & 0 & { - 1} \cr
0 & 1 & 0 \cr
} } \right|dxdy = \left( {{\bf{i}} + {\bf{k}}} \right)dxdy \cr
& dS = \left\| {d{\bf{S}}} \right\| = \sqrt 2 dxdy \cr
& {\bf{n}} = {1 \over {\sqrt 2 }}\left( {{\bf{i}} + {\bf{k}}} \right) \cr
& \nabla \times {\bf{F}} = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
{{\partial _x}} & {{\partial _y}} & {{\partial _z}} \cr
{xy} & {2z} & {6y} \cr
} } \right| = 4{\bf{i}} - x{\bf{k}} \cr} $$
I think you had a mistake in this part $d{\bf{S}}=dS {\bf{n}}$ where $\sqrt2$ cancels. Finally, the surface integral will be
$$\eqalign{
& I = \int\!\!\!\int {\nabla \times {\bf{F}} \cdot d{\bf{S}}} = \int_{x = - 6}^6 {\int_{y = - \sqrt {36 - {x^2}} }^{\sqrt {36 - {x^2}} } {\left( {4 - x} \right)dydx} } \cr
& \,\,\,\, = \int_{x = - 6}^6 {2\left( {4 - x} \right)\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = \int_{x = - 6}^6 {8\sqrt {36 - {x^2}} dx} = 8\int_{x = - 6}^6 {\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = 8\left( {18\pi } \right) = 144\pi \cr} $$
The result of the line integral is correct, so the mistake must be with your calculation of the surface integral. You have $\nabla \times \vec F = \left( 0,1-2xz,y^2 \right)$ and with a surface of the form $z=g(x,y)$, the normal $\vec n$ is given by $\left(-g_x,-g_y,1\right)$. The projection of the surface onto the $xy$-plane is a disc centered in the origin and with radius $\sqrt{6}$, so you have to integrate over this disc $D$:
$$\iint_D \left( 0,1-2xz,y^2 \right) \cdot \left(-g_x,-g_y,1\right) \,\mbox{d}A$$
where $z=g(x,y)$ with $g(x,y)=x^2+y^2$, so:
$$\iint_D \left( 0,1-2x\left(x^2+y^2\right),y^2 \right) \cdot \left(-2x,-2y,1\right) \,\mbox{d}A =
\iint_D \left( 4 x^3 y + 4 x y^3 + y^2 - 2 y \right)
\,\mbox{d}x\,\mbox{d}y$$
You can choose to switch to polar coordinates or not, but you should find $9\pi$ either way.
Maybe this is sufficient to find your mistake? If not, perhaps you can show us your calculations.
Best Answer
Method 1: Calculating the Line Integral.
Lookings at the plots of the surfaces $$ \color{blue}{z = y^2} \qquad \color{orange}{x^2+y^2=1} $$
We see that the intersecting curve $C$ looks like:
We can parametrize this curve by seeing that, since the $x,y$ coordinates lie on the unit circle, we have: $$C: \mathbf{x}(t) = \langle \cos t, \sin t, \sin^2 t \rangle \qquad (0 \leq t \leq 2\pi) $$
Taking the vector field: $$ \mathbf{F}(x,y,z) = \langle 2yz, xz, xy \rangle $$
We evaluate the line integral: \begin{align*} \int_C \mathbf{F} \cdot d\mathbf{s} &= \int_0^{2\pi} \mathbf{F}(\mathbf{x}(t)) \cdot \mathbf{x}'(t) \, dt \\ &= \int_0^{2\pi} \langle 2\sin^3 t, \cos t \sin^2 t, \cos t \sin t \rangle \cdot \langle -\sin t, \cos t, 2\sin t \cos t \rangle \, dt \\ &= \int_0^{2\pi} \left(3 \cos^2 t \sin^2 t - 2\sin^4 t \right) \, dt \\ &= \int_0^{2\pi} \left[3\left(\frac{\sin 2t}{2} \right)^2 - 2\left(\frac{1 - \cos 2t}{2} \right)^2 \right] \, dt \\ &= \int_0^{2\pi} \left[\frac{3\sin^2 2t}{4} - \frac{2 - 4\cos 2t + 2\cos^2 2t}{4} \right] \, dt \\ &= \int_0^{2\pi} \left[\frac{3\sin^2 2t - 2\cos^2 t + 4\cos 2t - 2}{4} \right] \, dt \\ &= \int_0^{2\pi} \left[\frac{\sin^2 2t + 3\cos 2t - 2}{4} \right] \, dt \\ &= \int_0^{2\pi} \left[\frac{1 - \cos 4t + 6\cos 2t - 4}{8} \right] \, dt \\ &= \int_0^{2\pi} \left[\frac{1 - 4}{8} \right] \, dt \\ &= -\frac{3}{8}\int_0^{2\pi} \, dt \\ &= -\frac{3\pi}{4} \end{align*}
Method 2: Applying Stokes' Theorem.
We must choose a surface $S$ that has $C$ as its boundary. We can simply choose the part of the surface $\color{blue}{z = y^2}$ thats enclosed within $\color{orange}{x^2+y^2=1}$, that is:
$$S = \{ (x,y,z) \mid x^2 + y^2 \leq 1 \,,\, z = y^2 \}$$
We can parametrize the surface by noting that the $(x,y)$ points form a solid disk in the $x$-$y$ space, and $z$ is simply $y^2$, so we parametrize the surface as
$$S: \quad \mathbf{X}(r,\theta) = \langle r \cos \theta, r \sin \theta, r^2 \sin^2 \theta \rangle \qquad (\underbrace{0 \leq r \leq 1, 0 \leq \theta \leq 2\pi}_{(r,\theta) \in \, R = [0,1] \times [0,2\pi]})$$
We find the normal vector that arises from this parametrization by calculating: \begin{align*} \frac{\partial \mathbf{X}}{\partial r} &= \langle \cos \theta, \sin \theta, 2r \sin^2 \theta \rangle \\ \frac{\partial \mathbf{X}}{\partial \theta} &= \langle -r \sin \theta, r\cos \theta, 2r^2 \sin \theta \cos \theta \rangle \\ \mathbf{N}(r,\theta) &= \frac{\partial \mathbf{X}}{\partial r} \times \frac{\partial \mathbf{X}}{\partial \theta} \\ &= \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ \cos \theta & \sin \theta & 2r \sin^2 \theta \\ -r \sin \theta & r\cos \theta & 2r^2 \sin \theta \cos \theta \end{vmatrix} \\ &= \langle 0, -2r^2 \sin \theta, r \rangle \end{align*} We note that the $z$ component of the normal vector is positive, so as to point upward, conforming with the counterclockwise orientation of the line integral taken before.
The curl of the vector field $\mathbf{F}$ is: \begin{align*} \nabla \times \mathbf{F} &= \begin{vmatrix} \mathbf{i} & \mathbf{j} & \mathbf{k} \\ \frac{\partial}{\partial x} & \frac{\partial}{\partial y} & \frac{\partial}{\partial z} \\ 2yz & xz & xy \end{vmatrix} = \langle 0, y, -z \rangle \end{align*}
And so, we evaluate the line integral over $C = \partial S$ by applying Stoke's Theorem: \begin{align*} \int_{\partial S} \mathbf{F} \cdot d\mathbf{s} &= \iint_S (\nabla \times \mathbf{F}) \cdot d\mathbf{S} \\ &= \iint_R (\nabla \times \mathbf{F})\mid_{\mathbf{X}(r,\theta)} \cdot \mathbf{N}(r,\theta) \, dA \\ &= \int_0^{2\pi} \int_0^1 \langle 0, r \sin \theta, -r^2 \sin^2 \theta \rangle \cdot \langle 0, -2r^2 \sin \theta, r \rangle \, dr \, d\theta \\ &= \int_0^{2\pi} \int_0^1 (-3r^3 \sin^2 \theta) \, dr \, d\theta \\ &= -\frac{3}{4}\int_0^{2\pi} \sin^2 \theta \, d\theta \\ &= -\frac{3}{8}\int_0^{2\pi} (1 - \cos 2\theta) \, d\theta \\ &= -\frac{3}{8}\int_0^{2\pi} 1 \, d\theta \\ &= -\frac{3\pi}{4} \end{align*}