To integrate this region in polar coordinates,
it is advisable to break up the integral into two parts,
as shown in the figures below:
The two parts of the integral are divided by the diagonal line through
the upper right corner of the rectangle.
Since the sides of the rectangle are $a$ and $b$,
this diagonal line is at the angle $\arctan \frac ba.$
For $0 \leq \theta \leq \arctan \frac ba,$
you would integrate over $0 \leq r \leq a \sec\theta,$
and for $\arctan \frac ba \leq \theta \leq \frac\pi2,$
you would integrate over $0 \leq r \leq b \csc\theta.$
If you actually try this, I think you'll find that it is no easier than
doing the integration in rectangular coordinates.
It may even be worse.
An alternative approach, rather than combining $x^2+y^2$ into $r^2$,
is to integrate the terms separately:
$$\begin{eqnarray}
m &=& \int_0^a\int_0^b (1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b dy\,dx
+\int_0^a\int_0^b x^2 \,dy\,dx
+\int_0^a\int_0^b y^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{x} &=& \int_0^a\int_0^b x(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b x \,dy\,dx
+\int_0^a\int_0^b x^3 \,dy\,dx
+\int_0^a\int_0^b xy^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{y} &=& \int_0^a\int_0^b y(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b y \,dy\,dx
+\int_0^a\int_0^b x^2y \,dy\,dx
+\int_0^a\int_0^b y^3 \,dy\,dx
\end{eqnarray}$$
Now you have nine integrals to solve, but they're all quite simple.
Actually, the factor of $2$ is unimportant. The center of mass in the $y$ direction is given by
$$\bar{y} = \frac{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)(r \sin{\theta})}{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)}$$
Note that the weight of the $y$ coordinate is expressed in the $r \sin{\theta}$ term in the numerator. Note also the boundary of the lamina is expressed in its polar form, $r=\sin{\theta}$. Evaluating the radial integrals, we get
$$\bar{y} = \frac{\frac{16}{4} \displaystyle \int_0^{\pi/2} d\theta \: \sin^5{\theta}}{\frac{8}{3} \displaystyle \int_0^{\pi/2} d\theta \: \sin^3{\theta}}$$
or
$$\bar{y} = \frac{6}{5}$$
For $\bar{x}$, we do a similar calculation:
$$\begin{align}\bar{x} &= \frac{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)(r \cos{\theta})}{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)}\\ &= \frac{\frac{16}{4} \displaystyle \int_0^{\pi/2} d\theta \: \cos{\theta} \sin^4{\theta}}{\frac{8}{3} \displaystyle \int_0^{\pi/2} d\theta \: \sin^3{\theta}}\\ &= \frac{9}{20}\end{align}$$
Best Answer
This integral can directly be performed $$\int_0^y \frac{x}{\sqrt{1-y^2}}dx=\frac{y^2}{2\sqrt{1-y^2}}$$ $$\int_0^{\frac12} \frac{y^2}{2\sqrt{1-y^2}}dy=\int_0^{\frac\pi6} \frac{\sin^2{\theta}}{2\sqrt{1-\sin^2{\theta}}} \cos{\theta} d\theta$$ $$=\int_0^{\frac\pi6} \frac{\sin^2{\theta}}{2}d\theta$$ $$=\int_0^{\frac\pi6} \frac{1-\cos{2\theta}}{4}d\theta$$ $$=\frac14\left[\theta-\frac12\sin{2\theta}\right]_0^{\frac\pi6}$$ $$=\frac{\pi}{24}-\frac{\sqrt{3}}{16}$$ Where I have used the substitution $y=\sin{\theta}$ in order to calculate the second integral.