The second method is correct. It should be clear that operations that preserve normalness don't necessarily preserve the integral... a simpler example than the one you cite is just multiplying by a constant.
So you don't want any normal vector, you want a particular one. Actually, since you're taking the length at the end of the day, you want a particular function of $\theta$ and $\phi$... the direction doesn't actually matter.
More specifically you want what's called the surface element, which (vaguely) tells you what you need to multiply $d\theta d\phi$ by to get the area of the small patch of surface $\theta_0< \theta < \theta_0 + d\theta$, $\phi_0< \phi < \phi_0 + d\phi$ near the point $\phi_0,\theta_0$.
The surface element is computed by method 2 above. The fact that it's correct has nothing to do with the fact that the cross product of the tangent vectors points normal to the surface and everything to do with the fact that its length is the area of the paralellogram formed by the tangent vectors.
1) I don’t understand why didn’t you calculate the curl directly. I got $\nabla \times F = (0,0,3)$. I am sorry but — is the two $R\sin\theta\cos\phi$ a typo?
2) The parametrization part seems incorrect. In the context, the hemisphere has a radius 3. So you should probably go with $ \Phi ( \phi) = \begin{pmatrix} -R\sin \phi \\ R\cos \phi \\ 0 \end{pmatrix} $ instead. Besides, it seems that you didn’t calculate the derivative in the last integration formula.
By the way, in both cases, my answer is $3\times \pi R^2=27\pi$.
EDIT: As a reply to the comment:
This is how I would work out the curve integral. First, find a parametrization such that it moves anti-clockwise as you see from the top. It has to rotate in that way because the Stokes’ theorem involves direction. For example, let $x=R\cos t$, $y=R\sin t$ and $z=z$, where $t\in [0,2\pi]$. Then use
$$ \oint_C F(x,y,z) \cdot\mathrm d l= \int_0^{2\pi} F(x,y,z) \cdot (x’(t), y’(t), z’(t)) \mathrm d t$$
Note that we have $\mathrm d l = (x’(t), y’(t), z’(t)) \mathrm d t$. In this case,
$$ \oint_C F(x,y,z) \cdot\mathrm d l = \int_0^{2\pi} (-R\sin t, 2R\cos t, 0)\cdot(-R\sin t, R\cos t, 0) \mathrm d x = R^2\int_0^{2\pi} (\sin^2 t + 2\cos^2 t) \mathrm d t = 3\pi R^2 $$
Best Answer
Alternatively apply divergence theorem.
$\displaystyle \iint_S x^2dydz+y^2dxdz+z^2dxdy = \iiint_V (div \vec{F}) \ dV$
$\vec{F} = (x^2,y^2,z^2)$
$div(\vec{F}) = \nabla \cdot \vec{F} = 2(x+y+z)$
Please use $x = a + \rho \cos \theta \sin\phi, y = b + \rho \sin\theta \sin\phi, z = c + \rho \cos\phi$ for the integral.