To integrate this region in polar coordinates,
it is advisable to break up the integral into two parts,
as shown in the figures below:
The two parts of the integral are divided by the diagonal line through
the upper right corner of the rectangle.
Since the sides of the rectangle are $a$ and $b$,
this diagonal line is at the angle $\arctan \frac ba.$
For $0 \leq \theta \leq \arctan \frac ba,$
you would integrate over $0 \leq r \leq a \sec\theta,$
and for $\arctan \frac ba \leq \theta \leq \frac\pi2,$
you would integrate over $0 \leq r \leq b \csc\theta.$
If you actually try this, I think you'll find that it is no easier than
doing the integration in rectangular coordinates.
It may even be worse.
An alternative approach, rather than combining $x^2+y^2$ into $r^2$,
is to integrate the terms separately:
$$\begin{eqnarray}
m &=& \int_0^a\int_0^b (1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b dy\,dx
+\int_0^a\int_0^b x^2 \,dy\,dx
+\int_0^a\int_0^b y^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{x} &=& \int_0^a\int_0^b x(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b x \,dy\,dx
+\int_0^a\int_0^b x^3 \,dy\,dx
+\int_0^a\int_0^b xy^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{y} &=& \int_0^a\int_0^b y(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b y \,dy\,dx
+\int_0^a\int_0^b x^2y \,dy\,dx
+\int_0^a\int_0^b y^3 \,dy\,dx
\end{eqnarray}$$
Now you have nine integrals to solve, but they're all quite simple.
I will assume that you have been supplied with a formula to compute surface area.
We want an expression for the equation of the cone. Really one should use cylindrical coordinates, but let's use the usual Cartesian setup.
Think of the cone as having axix the $z$-axis, and pointing down with the apex at the origin.
We find the radius $r_z$ of the circle of cross-section at height $z$.
By similar triangles we have
$$\frac{r_z}{z}=\frac{a}{h},$$
so $r_x=\frac{a}{h}z$.
Thus the surface of the cone satisfies the equation
$$x^2+y^2=\frac{a^2}{h^2}z^2.$$
Now compute $dS$ as usual, and integrate.
Remarks: $1.$ Actually, this is really a $1$-variable problem, since we are dealing with a surface of revolution. Thus the problem could also be solved using techniques from first-year calculus.
$2.$ In the unlikely case that you do not know a formula for surface area, note that if $(x,y)$ ranges over $R$, and our surface has equation $z=f(x,y)$, then the surface area is given by
$$\iint_{R} \sqrt{f_x^2(x,y)+f_y^2(x,y)+1}\,dx\,dy,$$
where $f_x$ and $f_y$ are the partial derivatives.
Best Answer
It should be $\frac 12a$ not $\frac 13a$
You can see that $$\int\cos\phi\sin\phi d\phi=\color{red}{\frac 12}\sin^{\color{red}{2}}\phi$$