Option 1:
The projection of the intersection of $z=x^2+y^2$ and $z=y$ in the $xy$ plane is $x^2+y^2=y$, i.e., the circle centered at $(0,1/2)$ with radius $1/2$, or in polar coordinates, $r=\sin \theta$. So you can parametrize the surface $S$ as follows:
$$
\begin{cases}
x=x \\
y=y \quad \quad \quad \quad \mbox{with }(x,y)\in D=\{(r,\theta)\;|\; 0 \le \theta \le \pi, \; 0 \le r \le \sin \theta \}\\
z=x^2+y^2
\end{cases}
$$
It follows that
$$
\Phi = \iint_S \nabla \times \vec{v} \; dS = \iint_D \nabla \times \vec{v}(x,y)\cdot \pmatrix{1\\0\\2x}\times \pmatrix{0\\1\\2y}\; dA = \iint_D \pmatrix{1\\2\\1} \cdot \pmatrix{-2x\\-2y\\1}\; dA \\= \iint_D 1-2x-4y \;dA
$$
This matches your answer, except for the $4$ coefficient for $y$. If I am not mistaken, $ \nabla \times \vec{v}=(1,2,1)$. Now what you need to do is integrate over $D$ (or $A$ with your notation), as defined above.
Switching to polar coordinates yields
$$
\Phi = \int_0^{\pi}\int_0^{\sin \theta}(1-2r \cos \theta-4r\sin\theta) r dr d\theta = -\frac{\pi}{4}
$$
Option 2:
Alternatively, if you are familiar with the divergence theorem:
$$
\Phi = \iint_S \nabla \times \vec{v} \; dS = \underbrace{\iiint_V \nabla\cdot \nabla \times \vec{v} \; dV}_{=0} - \iint_{S_2} \nabla \times \vec{v} \; dS
$$
where $S_2$ is the surface that closes $S$, i.e., the part of the plane $z=y$ on top of $D$, which we can parametrize with
$$
\begin{cases}
x=x \\
y=y \quad \quad \quad \quad \mbox{with }(x,y)\in D=\{(r,\theta)\;|\; 0 \le \theta \le \pi, \; 0 \le r \le \sin \theta \}\\
z=y
\end{cases}
$$
It follows that
$$
\Phi = - \iint_{S_2} \nabla \times \vec{v} \; dS = - \iint_D \nabla \times \vec{v}(x,y)\cdot \pmatrix{1\\0\\0}\times \pmatrix{0\\1\\1}\; dA = - \iint_D \pmatrix{1\\2\\1} \cdot \pmatrix{0\\-1\\1}\; dA = \iint_D dA = \frac{\pi}{4}
$$
When we use this theorem, we conventionally parametrize the surface "outwards", i.e., in the opposition direction as in Option $1$, hence the sign difference.
Option 3:
Last but not least, you can use Stoke's theorem:
$$
\Phi = \iint_S \nabla \times \vec{v} \; dS = \oint_C \vec{v} \cdot d\vec{r}
$$
where $C$ is the curve at the intersection between the paraboloid and the plane. Using the parametrization proposed by @levap in the question that you asked earlier :
$$
\begin{cases}
x=\frac{\cos t}{2} \\
y=\frac{1+\sin t}{2} \quad \quad \quad \quad \mbox{with } t\in [0,2\pi] \\
z=\frac{1+\sin t}{2}
\end{cases}
$$
It follows that
$$
\Phi = \oint_C \vec{v} \cdot d\vec{r} = \int_0^{2\pi} \frac{1}{4} \pmatrix{2(1+\sin t)\\ 1+ \cos t \\ 1 + \sin t}\cdot \pmatrix{-\sin t\\ \cos t \\ \cos t}\; dt = -\frac{\pi}{4}
$$
Isn't Calculus fun ? :)
Yes, it seems this should be converted to polar coordinates. We have a circular area, so it makes sense. The circle is at the origin with a radius of 4, so the bounds for $r$ are from $0$ to $4$. Then, because it's a complete circle, we integrate $\theta$ from $0$ to $2\pi$. Then, we have $z-xy=\pi$ that needs to be derived before being converted to polar coordinates:
$z=\pi+xy$
$z_x = y = r\sin(\theta), x_y = x = r\cos(\theta)$
Final integral: $\int_0^{2\pi} \int_0^4 \sqrt{1+(z_y)^2+(z_x)^2} =
\int_0^{2\pi} \int_0^4 \sqrt{1+r^2\sin^2(\theta)+r^2\cos^2(\theta)}$
$=\int_0^{2\pi} \int_0^4 \sqrt{1+r^2} \ dr d\theta$
Now here's some really helpful advice. This integral is about to turn ugly. You will have to do a trig substitution: Let $r = \tan(\alpha)$. Then $dr = \sec^2(\alpha)$ and we have $$\int_0^{2\pi} d\theta \cdot \int_{r=0}^{r=4} \sec^3(\alpha) \ d\alpha$$.
Have fun, and good luck!
Best Answer
Cylindrical coordinates look good, since $z = x^2 + y^2 = r^2$. The integral looks like $$ \iint_S \left( 1 + 4z \right)^3 \; dx \; dy = \int_0^{2\pi} \int_0^1 \left( 1 + 4r^2 \right)^3 r \; dr \; d\theta, $$ which can be computed with a substitution.