In fact, the only constraints for the vector $\bf{n}$ are
$1.$ The vector $\bf{n}$ is a unit vector normal to the surface.
$2.$ It should have proper orientation depending on the orientation of the surrounding curve.
So, I think you may have made a mistake in the problem you solved and hence we may help you if you write it down in your question. :)
Verifying Stokes Theorem For Your Question
Your surface is enclosed by the intersection curve of the plane $x+z=1$ and the cylinder $x^2+y^2=36$ as the following figure shows.
The parametric equation of the intersection curve, the tangent vector, and the vector field are
$$\eqalign{
& {\bf{x}} = 6\cos \theta {\bf{i}} + 6\sin \theta {\bf{j}} + \left( {1 - 6\cos \theta } \right){\bf{k}} \cr
& {{d{\bf{x}}} \over {d\theta }} = - 6\sin \theta {\bf{i}} + 6\cos \theta {\bf{j}} + 6\sin \theta {\bf{k}} \cr
& F({\bf{x}}) = xy{\bf{i}} + 2z{\bf{j}} + 6y{\bf{k}} \cr} $$
and hence the line integral will be
$$\eqalign{
& I = \int\limits_C {F({\bf{x}}) \cdot {{d{\bf{x}}} \over {d\theta }}d\theta } = \int_{\theta = 0}^{2\pi } {\left( { - 6\sin \theta xy + 12\cos \theta z + 36\sin \theta y} \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta + 2\cos \theta \left( {1 - 6\cos \theta } \right) + 36{{\sin }^2}\theta } \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta - 12{{\cos }^2}\theta + 36{{\sin }^2}\theta + 2\cos \theta } \right)d\theta } \cr
& \,\,\, = 6\left[ { - 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta \cos \theta d\theta } - 12\int_{\theta = 0}^{2\pi } {{{\cos }^2}\theta d\theta } + 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta d\theta + 2\int_{\theta = 0}^{2\pi } {\cos \theta d\theta } } } \right] \cr
& \,\,\, = 6\left[ { - 36\left( 0 \right) - 12\left( \pi \right) + 36\left( \pi \right) + 2\left( 0 \right)} \right] \cr
& \,\,\, = 144\pi \cr} $$
Next, compute the area element vector $d\bf{S}$ and $\nabla \times {\bf{F}}$
$$\eqalign{
& {\bf{x}} = x{\bf{i}} + y{\bf{j}} + \left( {1 - x} \right){\bf{k}} \cr
& d{\bf{S}} = \left( {{{\partial {\bf{x}}} \over {\partial x}} \times {{\partial {\bf{x}}} \over {\partial y}}} \right)dxdy = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
1 & 0 & { - 1} \cr
0 & 1 & 0 \cr
} } \right|dxdy = \left( {{\bf{i}} + {\bf{k}}} \right)dxdy \cr
& dS = \left\| {d{\bf{S}}} \right\| = \sqrt 2 dxdy \cr
& {\bf{n}} = {1 \over {\sqrt 2 }}\left( {{\bf{i}} + {\bf{k}}} \right) \cr
& \nabla \times {\bf{F}} = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
{{\partial _x}} & {{\partial _y}} & {{\partial _z}} \cr
{xy} & {2z} & {6y} \cr
} } \right| = 4{\bf{i}} - x{\bf{k}} \cr} $$
I think you had a mistake in this part $d{\bf{S}}=dS {\bf{n}}$ where $\sqrt2$ cancels. Finally, the surface integral will be
$$\eqalign{
& I = \int\!\!\!\int {\nabla \times {\bf{F}} \cdot d{\bf{S}}} = \int_{x = - 6}^6 {\int_{y = - \sqrt {36 - {x^2}} }^{\sqrt {36 - {x^2}} } {\left( {4 - x} \right)dydx} } \cr
& \,\,\,\, = \int_{x = - 6}^6 {2\left( {4 - x} \right)\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = \int_{x = - 6}^6 {8\sqrt {36 - {x^2}} dx} = 8\int_{x = - 6}^6 {\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = 8\left( {18\pi } \right) = 144\pi \cr} $$
Your doubts are more than justified. They've given you a problem with a contrived hypothesis that cannot exist. But you do seem to have a misunderstanding. You choose a surface whose boundary curve is $C$. One such example would be the disk $z=3$, $x^2+y^2\le 9$. The normal $\vec n$ to that surface is indeed $\vec k$, so the flux of the curl will be $0$.
But what if you choose the original cone $S$ (and don't worry about the sharp point at $z=0$)? Then you can compute and get the $18\pi$.
How can you get two different answers? Easily, because they gave you a curl field that cannot possible be a curl. It's $\text{div}(\text{curl}\vec F) = 0$ that says this flux will not depend on the surface you pick with boundary curve $C$.
Best Answer
$\iint \nabla \times F\ dA = \oint f\cdot dr$
$\nabla \times F = (\frac{\partial}{\partial y} xy - \frac{\partial}{\partial z} x^2,\frac{\partial}{\partial z} 2y- \frac{\partial}{\partial x} xy,\frac{\partial}{\partial x} x^2 - \frac{\partial}{\partial y} 2y) = (x, -y, 2x-2)$
..... you can skip much of what follows, I am complicating things more than I need to ....
the normal to the surface is $(\frac {1}{\sqrt 3},\frac 1{\sqrt 3},\frac 1{\sqrt 3})$
$\iint \frac {3x - y - 2}{\sqrt 3} \ dS$
Parameterize that surface?
$u = \frac {x-y}{\sqrt 2}\\ v = \frac {x+y - 2z}{\sqrt 6} \\ w = \frac {x+y+z}{\sqrt 3}$
This coordinate transformation is orthonormal.
jacobian $\ dx\ dy\ dz = du\ dv\ dw$
In these coordinates $w = 0$
The semi-major axis of this elipse is at the point $(0,\sqrt {3},0)$ and the semi-minor axis is at $(1,0,0)$
or
$x = \frac{1}{\sqrt 2} u + \frac {1}{\sqrt 6} v + \frac{1}{\sqrt 3} w\\ y = -\frac{1}{\sqrt 2} u + \frac {1}{\sqrt 6} v + \frac{1}{\sqrt 3} w\\ z = -\frac {2}{\sqrt 6} v + \frac{1}{\sqrt 3} w$
$x^2 + y^2 = 1$ with $x+y+z = 0$ becomes $u^2 + \frac {v^2}3 = 1$ with $w= 0$
Putting it together we have
$\iint \frac {a_1 u + a_2 v - 2}{\sqrt 3} \ du \ dv$
We don't need to work out what the linear coefficients of u,v are because whatever they are, over this ellipse:
$\iint a_1 u + a_2 v\ du\ dv = 0$
$-\frac 2{\sqrt 3} A$
$A$ is the area of the ellipse. $A = \pi\sqrt 2$
The integral evaluates to $-2\pi$
.... Resume here, I don't have the heart do delete it .......
instead of the coordinate transformation, we can say:
$x = x\\y = z\\ z= -x-y\\dS = (-\frac {\partial z}{\partial x},-\frac {\partial z}{\partial y}, 1)\ dx\ dy$
$\iint (x, -y, 2x-2)\cdot (1,1,1)\ dx\ dy\\ \iint x -y+ 2x-2\ dx\ dy$
$\iint x -y+ 2x \ dx\ dy$ are odd functions evaluated over a region that is symmetric.
$\iint -2 \ dx\ dy = -2\pi$
How did this compare to calculating it directly?
$(x,y,z) = (\cos t, \sin t, 1 - \cos t - \sin t) dr = (-\sin t, \cos t, 1 + \sin t - \cos t)$
$F\cdot dr = (2\sin t, \cos^2 t, \cos t\sin t)\cdot(-\sin t, \cos t, 1 + \sin t - \cos t)\\ -2\sin^2t + \cos^3 t + \cos t\sin t + \cos t\sin^2 t - \cos^2 t\sin t$
That is going to be a snap to integrate. Most of those terms evaluate to 0
$\int_0^{2\pi} -2\sin^2t + \cos^3 t + \cos t\sin t + \cos t\sin^2 t - \cos^2 t\sin t \ dt\\ \int_0^{2\pi} -2\sin^2t\ dt\\ -2\pi$