Using Stokes' theorem the integral is
$$I =\int_S \text{curl } \mathbf{F} \cdot \mathbf{n} \,dS = \oint_C \mathbf{F} \cdot d \,\mathbf{l} $$
where $C$ is the circle of radius $r$ centered at the origin of the $xy-$plane.
On $C$ we have, using polar coordinates, $F_x = xy^2 \tanh(x^2) = r^3\cos \theta\sin^2 \theta \tanh(r^2 \cos^2 \theta)$ and $F_y = x = r \cos \theta$. The unit tangent to $C$ is $- \sin \theta \, \mathbf{e}_x + \cos \theta \, \mathbf{e}_y$, with no $z-$component. Consequently,
$$I = \int_0^{2\pi} r^3\cos\theta \sin^2 \theta \tanh(r^2 \cos^2 \theta) (-\sin \theta) r \, d \theta + \int_0^{2 \pi}r \cos \theta (\cos \theta) r\,d \theta \\ = -r^4 \underbrace{\int_0^{2\pi}\cos \theta \sin^3 \theta \tanh(r^2\cos^2 \theta) \, d \theta}_{= \,0 \text{ by symmetry}} \, + \,r^2\int_0^{2\pi}\cos^2 \theta \, d \theta \\ = \pi r^2$$
For $\textbf{r} = (u+v) \textbf{i} + (u-v)\textbf{j} + (1-2u)\textbf{k}$ to fill the triangle with vertices $(1,0,0)$, $(0,1,0)$, and $(0,0,1)$ the following restrictions must be placed on the components of $\textbf{r}$
\begin{equation} \tag{1}
0 \leq u+v \leq 1
\end{equation}
\begin{align} \tag{2}
0 \leq u-v \leq 1
\end{align}
\begin{align} \tag{3}
0 \leq 1-2u \leq 1
\end{align}
Subtracting $(3)$ from $(1)$
$ \implies 0\leq u+v - (1-2u) \leq 0 \implies 3u-1-v =0 $
$$ v=3u-1 \tag{4}$$
Substitution of $(4)$ into either $(1)$ or $(2)$ shows that $$0 \leq u \leq \frac{1}{2} \tag{5}$$
Notice that $(2)$ implies $v \leq u$. Therefore, for all $u$ that satisfy $(5)$, we have
$$ 3u-1\leq v \leq u \tag{6} $$
Hence $(5)$ and $(6)$ are the bounds for $u$ and $v$
Note: You want $\textbf{n}$ to be in the opposite direction of $-2(1,1,1)$ since this has a negative $z$-component
Edit
The range for $v$ I originally provided was wrong, but the above ranges should be correct. This can be shown by using the fact that surface integrals are independent under a change of parametric representation. Consider a second parameterization
$$\textbf{r}(x,y) = x \textbf{i} + y \textbf{i} + (1-x-y) \textbf{i} $$
this gives a normal
$\textbf{n} = \frac{\partial \textbf{r} }{\partial x} \times \frac{\partial \textbf{r} }{\partial y} = \textbf{i} + \textbf{j} + \textbf{k} $.
Then
$$ \iint_S \textbf{F} \cdot \textbf{n} \, dS = \iint_T (x,y,1-x-y) \cdot (1,1,1) dx\, dy = \iint_T dx \, dy = \int_{0}^{1-x} \int_{0}^{1} dx\, dy = \frac{1}{2} $$
Now the parameterization
$\textbf{r} = (u+v) \textbf{i} + (u-v)\textbf{j} + (1-2u)\textbf{k}$ together with $\textbf{n} = 2\textbf{i} + 2\textbf{j} + 2\textbf{k}$ and the bounds for $u$ and $v$ must give the same answer
$$\iint_S \textbf{F} \cdot \textbf{n} \, dS = \iint_T (u+v,u-v,1-2u) \cdot (2,2,2) du\, dv = 2 \iint_T du\, dv = 2 \int_{0}^{\frac{1}{2}} \int_{3u-1}^{u} dv\, du = \frac{1}{2} $$
Best Answer
Using divergence theorem, one can easily see that the result is volume of the unit sphere which is $\frac{4\pi}{3}$. But as the question specifically seeks to use representation $z=\sqrt{1-x^2-y^2}$ and do surface integral,
use the fact that radially outward normal vector for a sphere is $(x,y,z)$ which can be rewritten as $(\frac{x}{\sqrt{1-x^2-y^2}}, \frac{y}{\sqrt{1-x^2-y^2}}, 1)$ using the representation $z = \sqrt{1-x^2-y^2}$.
Or you can take partial derivative as below,
$-\displaystyle \frac{\partial z}{\partial x} = \frac{x}{\sqrt{1-x^2-y^2}}$
$-\displaystyle \frac{\partial z}{\partial y} = \frac{y}{\sqrt{1-x^2-y^2}}$
So the normal vector, $\vec n = (\frac{x}{\sqrt{1-x^2-y^2}}, \frac{y}{\sqrt{1-x^2-y^2}}, 1)$
$\vec F = (x,y,0)$
$\vec F \cdot \vec n = \frac{x^2 + y^2}{\sqrt{1-x^2-y^2}}$
Surface integral should be
$\displaystyle \int_{-1}^{1} \int_{-\sqrt{1-y^2}}^{\sqrt{1-y^2}} \frac{x^2 + y^2}{\sqrt{1-x^2-y^2}} \ dx \ dy$