In fact, the only constraints for the vector $\bf{n}$ are
$1.$ The vector $\bf{n}$ is a unit vector normal to the surface.
$2.$ It should have proper orientation depending on the orientation of the surrounding curve.
So, I think you may have made a mistake in the problem you solved and hence we may help you if you write it down in your question. :)
Verifying Stokes Theorem For Your Question
Your surface is enclosed by the intersection curve of the plane $x+z=1$ and the cylinder $x^2+y^2=36$ as the following figure shows.
The parametric equation of the intersection curve, the tangent vector, and the vector field are
$$\eqalign{
& {\bf{x}} = 6\cos \theta {\bf{i}} + 6\sin \theta {\bf{j}} + \left( {1 - 6\cos \theta } \right){\bf{k}} \cr
& {{d{\bf{x}}} \over {d\theta }} = - 6\sin \theta {\bf{i}} + 6\cos \theta {\bf{j}} + 6\sin \theta {\bf{k}} \cr
& F({\bf{x}}) = xy{\bf{i}} + 2z{\bf{j}} + 6y{\bf{k}} \cr} $$
and hence the line integral will be
$$\eqalign{
& I = \int\limits_C {F({\bf{x}}) \cdot {{d{\bf{x}}} \over {d\theta }}d\theta } = \int_{\theta = 0}^{2\pi } {\left( { - 6\sin \theta xy + 12\cos \theta z + 36\sin \theta y} \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta + 2\cos \theta \left( {1 - 6\cos \theta } \right) + 36{{\sin }^2}\theta } \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta - 12{{\cos }^2}\theta + 36{{\sin }^2}\theta + 2\cos \theta } \right)d\theta } \cr
& \,\,\, = 6\left[ { - 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta \cos \theta d\theta } - 12\int_{\theta = 0}^{2\pi } {{{\cos }^2}\theta d\theta } + 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta d\theta + 2\int_{\theta = 0}^{2\pi } {\cos \theta d\theta } } } \right] \cr
& \,\,\, = 6\left[ { - 36\left( 0 \right) - 12\left( \pi \right) + 36\left( \pi \right) + 2\left( 0 \right)} \right] \cr
& \,\,\, = 144\pi \cr} $$
Next, compute the area element vector $d\bf{S}$ and $\nabla \times {\bf{F}}$
$$\eqalign{
& {\bf{x}} = x{\bf{i}} + y{\bf{j}} + \left( {1 - x} \right){\bf{k}} \cr
& d{\bf{S}} = \left( {{{\partial {\bf{x}}} \over {\partial x}} \times {{\partial {\bf{x}}} \over {\partial y}}} \right)dxdy = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
1 & 0 & { - 1} \cr
0 & 1 & 0 \cr
} } \right|dxdy = \left( {{\bf{i}} + {\bf{k}}} \right)dxdy \cr
& dS = \left\| {d{\bf{S}}} \right\| = \sqrt 2 dxdy \cr
& {\bf{n}} = {1 \over {\sqrt 2 }}\left( {{\bf{i}} + {\bf{k}}} \right) \cr
& \nabla \times {\bf{F}} = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
{{\partial _x}} & {{\partial _y}} & {{\partial _z}} \cr
{xy} & {2z} & {6y} \cr
} } \right| = 4{\bf{i}} - x{\bf{k}} \cr} $$
I think you had a mistake in this part $d{\bf{S}}=dS {\bf{n}}$ where $\sqrt2$ cancels. Finally, the surface integral will be
$$\eqalign{
& I = \int\!\!\!\int {\nabla \times {\bf{F}} \cdot d{\bf{S}}} = \int_{x = - 6}^6 {\int_{y = - \sqrt {36 - {x^2}} }^{\sqrt {36 - {x^2}} } {\left( {4 - x} \right)dydx} } \cr
& \,\,\,\, = \int_{x = - 6}^6 {2\left( {4 - x} \right)\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = \int_{x = - 6}^6 {8\sqrt {36 - {x^2}} dx} = 8\int_{x = - 6}^6 {\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = 8\left( {18\pi } \right) = 144\pi \cr} $$
The result of the line integral is correct, so the mistake must be with your calculation of the surface integral. You have $\nabla \times \vec F = \left( 0,1-2xz,y^2 \right)$ and with a surface of the form $z=g(x,y)$, the normal $\vec n$ is given by $\left(-g_x,-g_y,1\right)$. The projection of the surface onto the $xy$-plane is a disc centered in the origin and with radius $\sqrt{6}$, so you have to integrate over this disc $D$:
$$\iint_D \left( 0,1-2xz,y^2 \right) \cdot \left(-g_x,-g_y,1\right) \,\mbox{d}A$$
where $z=g(x,y)$ with $g(x,y)=x^2+y^2$, so:
$$\iint_D \left( 0,1-2x\left(x^2+y^2\right),y^2 \right) \cdot \left(-2x,-2y,1\right) \,\mbox{d}A =
\iint_D \left( 4 x^3 y + 4 x y^3 + y^2 - 2 y \right)
\,\mbox{d}x\,\mbox{d}y$$
You can choose to switch to polar coordinates or not, but you should find $9\pi$ either way.
Maybe this is sufficient to find your mistake? If not, perhaps you can show us your calculations.
Best Answer
$\oint_c \vec F \cdot d\vec r = \oint_c -2yz\,dx + y\,dy + 3x\,dz = \oint_c -2yz\,dx + y\,dy = \iint (\frac{\partial}{\partial x}(y) - \frac{\partial}{\partial y}(-2yz))dA$
Okay you have used Green's theorem here which is the baby version of stokes theorem when your contour lies in a plane.
continuing.
$\iint 2z\ dA$
$A$ is the disk of radius $2,$ at $z = 1$
$8\pi$
If we don't use Green's theorem.
$x = 2 \cos t\\ y = 2\sin t\\ z = 1$
$\int_0^{2\pi} 8\sin^2 t + 4\sin t\cos t \ dt = 8\pi$
Stokes theorem.
$dS = (-\frac {\partial z}{\partial x},-\frac {\partial z}{\partial y}, 1) = (2x, 2y, 1)$
(it took me far too long to spot this error)
$\iint \nabla \times F dS = \iint -4y^2 - 6y + 2(5-x^2 - y^2) \ dy\ dx$
Converting to polar seem to make sense here, but is not entirely necessary.
$\int_0^{2\pi}\int_0^2 -4r^3 \sin^2\theta - 6r^2\sin\theta + 10r - 2r^2\ dr\ d\theta$
Lets integrate by $\theta$ first.
$\int_0^2 20\pi r -8\pi r^3 \ dr\\ 10 r^2 - \frac {8}{4} r^4|_0^2\pi\\ 8\pi$
By the way, one of the implications of Stokes theorem, is that divergence free force integrated over any surface with the same boundary has the same flux.
So the integral over the disk equals the integral over the cone.