In fact, the only constraints for the vector $\bf{n}$ are
$1.$ The vector $\bf{n}$ is a unit vector normal to the surface.
$2.$ It should have proper orientation depending on the orientation of the surrounding curve.
So, I think you may have made a mistake in the problem you solved and hence we may help you if you write it down in your question. :)
Verifying Stokes Theorem For Your Question
Your surface is enclosed by the intersection curve of the plane $x+z=1$ and the cylinder $x^2+y^2=36$ as the following figure shows.
The parametric equation of the intersection curve, the tangent vector, and the vector field are
$$\eqalign{
& {\bf{x}} = 6\cos \theta {\bf{i}} + 6\sin \theta {\bf{j}} + \left( {1 - 6\cos \theta } \right){\bf{k}} \cr
& {{d{\bf{x}}} \over {d\theta }} = - 6\sin \theta {\bf{i}} + 6\cos \theta {\bf{j}} + 6\sin \theta {\bf{k}} \cr
& F({\bf{x}}) = xy{\bf{i}} + 2z{\bf{j}} + 6y{\bf{k}} \cr} $$
and hence the line integral will be
$$\eqalign{
& I = \int\limits_C {F({\bf{x}}) \cdot {{d{\bf{x}}} \over {d\theta }}d\theta } = \int_{\theta = 0}^{2\pi } {\left( { - 6\sin \theta xy + 12\cos \theta z + 36\sin \theta y} \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta + 2\cos \theta \left( {1 - 6\cos \theta } \right) + 36{{\sin }^2}\theta } \right)d\theta } \cr
& \,\,\, = 6\int_{\theta = 0}^{2\pi } {\left( { - 36{{\sin }^2}\theta \cos \theta - 12{{\cos }^2}\theta + 36{{\sin }^2}\theta + 2\cos \theta } \right)d\theta } \cr
& \,\,\, = 6\left[ { - 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta \cos \theta d\theta } - 12\int_{\theta = 0}^{2\pi } {{{\cos }^2}\theta d\theta } + 36\int_{\theta = 0}^{2\pi } {{{\sin }^2}\theta d\theta + 2\int_{\theta = 0}^{2\pi } {\cos \theta d\theta } } } \right] \cr
& \,\,\, = 6\left[ { - 36\left( 0 \right) - 12\left( \pi \right) + 36\left( \pi \right) + 2\left( 0 \right)} \right] \cr
& \,\,\, = 144\pi \cr} $$
Next, compute the area element vector $d\bf{S}$ and $\nabla \times {\bf{F}}$
$$\eqalign{
& {\bf{x}} = x{\bf{i}} + y{\bf{j}} + \left( {1 - x} \right){\bf{k}} \cr
& d{\bf{S}} = \left( {{{\partial {\bf{x}}} \over {\partial x}} \times {{\partial {\bf{x}}} \over {\partial y}}} \right)dxdy = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
1 & 0 & { - 1} \cr
0 & 1 & 0 \cr
} } \right|dxdy = \left( {{\bf{i}} + {\bf{k}}} \right)dxdy \cr
& dS = \left\| {d{\bf{S}}} \right\| = \sqrt 2 dxdy \cr
& {\bf{n}} = {1 \over {\sqrt 2 }}\left( {{\bf{i}} + {\bf{k}}} \right) \cr
& \nabla \times {\bf{F}} = \left| {\matrix{
{\bf{i}} & {\bf{j}} & {\bf{k}} \cr
{{\partial _x}} & {{\partial _y}} & {{\partial _z}} \cr
{xy} & {2z} & {6y} \cr
} } \right| = 4{\bf{i}} - x{\bf{k}} \cr} $$
I think you had a mistake in this part $d{\bf{S}}=dS {\bf{n}}$ where $\sqrt2$ cancels. Finally, the surface integral will be
$$\eqalign{
& I = \int\!\!\!\int {\nabla \times {\bf{F}} \cdot d{\bf{S}}} = \int_{x = - 6}^6 {\int_{y = - \sqrt {36 - {x^2}} }^{\sqrt {36 - {x^2}} } {\left( {4 - x} \right)dydx} } \cr
& \,\,\,\, = \int_{x = - 6}^6 {2\left( {4 - x} \right)\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = \int_{x = - 6}^6 {8\sqrt {36 - {x^2}} dx} = 8\int_{x = - 6}^6 {\sqrt {36 - {x^2}} dx} \cr
& \,\,\,\, = 8\left( {18\pi } \right) = 144\pi \cr} $$
Best Answer
The paraboloid $z=x^2+y^2$ and the plane $z=ax+by$ intersect for
$$x^2+y^2=ax+by\implies\left(x-\frac a2\right)^2+\left(y-\frac b2\right)^2=\frac{a^2+b^2}4$$
i.e. in the cylinder with cross sections parallel to the $x$-$y$ plane centered at $\left(\frac a2,\frac b2\right)$ with radius $\frac{\sqrt{a^2+b^2}}2$. Naturally, I think this suggests a parametrization of $S$ in cylindrical coordinates,
$$\hat s(u,v)=x(u,v)\,\hat\imath+y(u,v)\,\hat\jmath+z(u,v)\,\hat k$$
where
$$\begin{cases}x(u,v)=u\cos v+\frac a2\\[1ex]y(u,v)=u\sin v+\frac b2\\[1ex]z(u,v)=x(u,v)^2+y(u,v)^2=\frac{a^2+b^2}4+u^2+au\cos v+bu\sin v\end{cases}$$
with $0\le u\le\frac{\sqrt{a^2+b^2}}2$ and $0\le v\le2\pi$.
From the integrand we can obtain the underlying vector field $\hat F$:
$$y^2\,\mathrm dz=\underbrace{(0\,\hat\imath+0\,\hat\jmath+y^2\,\hat k)}_{\hat F(x,y,z)}\cdot(\mathrm dx\,\hat\imath+\mathrm dy\,\hat\jmath+\mathrm dz\,\hat k)$$
Compute the curl:
$$\nabla\times\hat F=2y\,\hat\imath$$
Take the normal vector to $S$ to be $\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}$:
$$\hat n=\frac{\partial\hat s}{\partial v}\times\frac{\partial\hat s}{\partial u}=(au+2u^2\cos v)\,\hat\imath+(bu+2u^2\sin v)\,\hat\jmath-u\,\hat k$$
Then by Stokes' theorem the integral is
$$\begin{align*} \oint_Cy^2\,\mathrm dz&=\oint_C\hat F\cdot\mathrm d\hat r\\[1ex] &=\iint_S(\nabla\times\hat F)\cdot\mathrm d\hat S\\[1ex] &=\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}2}\left(2y(u,v)\,\hat\imath\right)\cdot\hat n\,\mathrm du\,\mathrm dv\\[1ex] &=2\int_0^{2\pi}\int_0^{\sqrt{a^2+b^2}/2}\left(u\sin v+\frac b2\right)\,\mathrm du\,\mathrm dv\\[1ex] &=\boxed{2\pi b\sqrt{a^2+b^2}} \end{align*}$$