Let $S$ be the surface of the hemisphere of radius $a$, and let $D$ be the disk underneath it in the $xy$-plane. Then we can consider the hemisphere as the graph of a function $f(x, y, z(x,y)) = z(x,y)$, where $z(x,y) = \sqrt{a^{2} - x^{2} - y^{2}}$. The surface integral of $S$ is:
$$
\iint\limits_{S}{{f\left( {x,y,z} \right)\,dS}} = \iint\limits_{D}{{f\left( {x,y,z\left( {x,y} \right)} \right)\sqrt {{{\left( {\frac{{\partial z}}{{\partial x}}} \right)}^2} + {{\left( {\frac{{\partial z}}{{\partial y}}} \right)}^2} + 1} \,dA}},
$$
where $dA$ is the area element of $D$. Then like you calculated,
$$
\frac{\partial z}{\partial x} = \frac{-x}{\sqrt{a^{2}-x^2-y^2}} = -\frac{x}{z},\qquad \frac{\partial z}{\partial y} = \frac{-y}{\sqrt{a^{2}-x^2-y^2}} = -\frac{y}{z},
$$
so
$$
\sqrt {{{\left( {\frac{{\partial z}}{{\partial x}}} \right)}^2} + {{\left( {\frac{{\partial z}}{{\partial y}}} \right)}^2} + 1} = \frac{1}{z}\bigg(x^{2} + y^{2} + z^{2}\bigg)^{\tfrac{1}{2}} = \frac{a}{z}.
$$
Substituting this in yields
$$
\iint\limits_{S}{{f\left( {x,y,z} \right)\,dS}} = \iint\limits_{D}{{z\cdot\frac{a}{z} \,dA}} = a\iint\limits_{D}{dA}.
$$
In spherical polar coordinates, $(x,y) = (a\sin\theta\cos\phi, a\sin\theta\sin\phi)$, for $0 \leq \theta < \tfrac{\pi}{2}$, $0 \leq \phi < 2\pi$. So as $dA = dx\, dy$, then
$$
dA = dx\, dy = \det\frac{\partial (x, y)}{\partial(\theta,\phi)} d\theta\, d\phi = a^{2}\sin\theta\cos\theta d\theta\, d\phi = \frac{a^{2}}{2} \sin(2\theta) d\theta\, d\phi,
$$
and consequently
$$
a\iint\limits_{D}{dA} = \frac{a^{3}}{2}\int_{0}^{\frac{\pi}{2}} \sin(2\theta) d\theta\, \int_{0}^{2\pi} d\phi = \frac{a^{3}}{2} \cdot \bigg[\frac{-1}{2}(-1 -1 )\bigg] \cdot 2\pi = a^{3}\pi,
$$
as the expected answer should be. Comparing this to your attempt, it looks like the mistake occurs when changing from the Cartesian area element to the spherical polar are element.
Using polar coordinates is not the best way.
Make the change of variable $u=x+y,v=x$. The integral becomes $\int_0^{1}\int_0^{u} e^{u^{2}} dvdu=\int_0^{1} ue^{u^{2}} du=\frac 1 2 e^{u^{2}}|_0^{1}=\frac 1 2 (e-1)$.
The conditions $0\leq x \leq 1,0\leq y \leq 1-x$ are equivalent to the condition $0\leq v \leq u \leq 1$.
Best Answer
Yes, using polar coordinates is a good idea. We find $$\iint_T\frac{dxdy}{(1+x^2+y^2)^2}=\int_{\theta=0}^{\pi/3}d\theta\int_{\rho=0}^{f(\theta)}\frac{\rho d\rho}{(1+\rho^2)^2} =-\frac{1}{2}\int_{\theta=0}^{\pi/3}\left[\frac{1}{1+\rho^2 }\right]_{\rho=0}^{f(\theta)}\,d\theta$$ where the upperbound $\rho=f(\theta)$ can be obtained from the line joining the points $(1,\sqrt{3})$ and $(2,0)$, $$\rho\sin(\theta)=y=\sqrt{3}(2-x)=\sqrt{3}(2-\rho\cos(\theta))$$ and therefore $$\rho=f(\theta)=\frac{2\sqrt{3}}{\sin(\theta)+\sqrt{3}\cos(\theta)} =\frac{\sqrt{3}}{\sin(\theta+\pi/3)}.$$ Can you take it from here?