Solution
Approach 1: Surface Integrals
For $S_1$, the curved surface projected over $xy$ plane
$$dS=\frac{dx\,dy}{\hat{n}\cdot\,\hat{k}}$$
and
$$\hat{n}=\frac{\nabla\cdot S}{|\nabla\cdot S|}=\frac{2x\,\hat{i}+2z\,\hat{k}}{\sqrt{4x^2+4z^2}}=\frac{2x\,\hat{i}+2z\,\hat{k}}{2\cdot3}=\frac{2x\,\hat{i}+2z\,\hat{k}}{6}=\frac{1}{3}x\,\hat{i}+\frac{1}{3}z\,\hat{k}$$
$$\hat{n}\cdot\hat{k}=\frac{1}{3}z$$
So,
$$dS=\frac{dx\,dy}{\frac{1}{3}z}=\frac{3\,dx\,dy}{z}$$
Now evaluating $\vec{A}\cdot \hat{n}$
$$\vec{A}\cdot \hat{n}=\left(6z\,\hat{i}+(2x+y)\,\hat{j}-x\,\hat{k}\right)\cdot\left(\frac{1}{3}x\,\hat{i}+\frac{1}{3}z\,\hat{k}\right) $$
$$\vec{A}\cdot \hat{n}=2zx-\frac{xz}{3}=\frac{5zx}{3}$$
$$S_1=\iint\limits_S \vec{A} \cdot \hat{n}\,dS=\frac{5xz}{3}\times \frac{3\,dy\,dx}{z}=\int\limits_0^8\int\limits_0^3 5x\, dx\, dy=180$$
$$S_1=180$$
For $S_2$ horizontal surface on $xy$ plane
$$\hat{n}=\hat{-k}$$
$$dS=dx\,dy$$
$$\vec{A}\cdot\hat{n}=x$$
$$S_2=\int\limits_0^8\int\limits_0^3 x\, dx\, dy=36$$
$$S_2=36$$
For $S_3$ flat surface on $yz$ plane
$$\hat{n}=\hat{-i}$$
$$dS=dy\,dz$$
$$\vec{A}\cdot\hat{n}=-6z$$
$$S_3=\int\limits_0^3\int\limits_0^8 6z\, dy\, dz=-216$$
$$S_3=-216$$
For $S_4$ flat 'quarter circle' surface on $xz$ plane
$$\hat{n}=\hat{-j}$$
$$dS=dx\,dz$$
$$\vec{A}\cdot\hat{n}=-(2x+y)$$
$$S_4=\int\limits_0^{\sqrt{9-x^2}}\int\limits_0^3 -(2x+y)\, dx\, dz$$
Here $y=0$
$$S_4=\int\limits_0^{\sqrt{9-x^2}}\int\limits_0^3 -2x\, dx\, dz$$
$$S_4=\int\limits_0^{\pi/2}\int\limits_0^3 -2r^2\cos{\theta} \, dr\, d\theta$$
Conversion to Polar coordinates
$dx\,dz=r\,dr\,d\theta$, $x=r\cos{\theta}$, $z=r\sin{\theta}$
$$S_4=\int\limits_0^{\pi/2}\int\limits_0^3 -2r^2\cos{\theta} \, dr\, d\theta$$
$$S_4=-18$$
For $S_5$ flat 'quarter circle' surface on $y=8$ plane
$$\hat{n}=\hat{j}$$
$$dS=dx\,dz$$
$$\vec{A}\cdot\hat{n}=(2x+y)$$
$$S_5=\int\limits_0^{\sqrt{9-x^2}}\int\limits_0^3 (2x+y)\, dx\, dz$$
Here $y=8$
$$S_5=\int\limits_0^{\sqrt{9-x^2}}\int\limits_0^3 (2x+8)\, dx\, dz$$
Conversion to Polar coordinates
$dx\,dz=r\,dr\,d\theta$, $x=r\cos{\theta}$, $z=r\sin{\theta}$
$$S_5=\int\limits_0^{\pi/2}\int\limits_0^3 (2r\cos{\theta}+8)\, r \, dr\, d\theta$$
$$S_5=\int\limits_0^{\pi/2}\int\limits_0^3 (2r^2\cos{\theta}+8r) \, dr\, d\theta$$
$$S_5=18(1+\pi)$$
Total
$$S=S_1+S_2+S_3+S_5=18\pi$$
Approach 2: Divergence Theorem
The Divergence Theorem
$$\iint\limits_S \vec{A}\cdot n\, dS=\iiint\limits_V \nabla\cdot\vec{A}\,\, dV$$
$$\nabla\cdot\vec{A}=1$$
$$dV=dx\,dy\,dz$$
$$S=\iiint_0^8\, dy\,dx\, dz=\iint 8\, dx\,dz$$
Conversion to Polar Coordinates
$$S=\int_0^{\pi/2}\int_0^3 8r\,dr\,d\theta=\int_0^{\pi/2}36\,d\theta=\left.36\,\theta\right|_0^{\pi/2}$$
$$S=18\pi$$
Best Answer
The small problem is that $\vec n$ needs to be normalized. But your bigger problem is that you are calculating the integral on the wrong surface. When you integrate $r$ from $0$ to $a$, and $\theta$ from $0$ to $2\pi$ (not $4\pi$), you are calculating the integral on the bottom cap of the cylinder, not on the side. So solving the first issue, $$\vec n=\frac{1}{2\sqrt{x^2+y^2}}(2x,2y,0)$$ Then the integrand will be $1$. For the second issue, the first integral is along the circumference, $dl$ from $0$ to $2\pi a$, and $dz$ from $0$ to $h$