To integrate this region in polar coordinates,
it is advisable to break up the integral into two parts,
as shown in the figures below:
The two parts of the integral are divided by the diagonal line through
the upper right corner of the rectangle.
Since the sides of the rectangle are $a$ and $b$,
this diagonal line is at the angle $\arctan \frac ba.$
For $0 \leq \theta \leq \arctan \frac ba,$
you would integrate over $0 \leq r \leq a \sec\theta,$
and for $\arctan \frac ba \leq \theta \leq \frac\pi2,$
you would integrate over $0 \leq r \leq b \csc\theta.$
If you actually try this, I think you'll find that it is no easier than
doing the integration in rectangular coordinates.
It may even be worse.
An alternative approach, rather than combining $x^2+y^2$ into $r^2$,
is to integrate the terms separately:
$$\begin{eqnarray}
m &=& \int_0^a\int_0^b (1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b dy\,dx
+\int_0^a\int_0^b x^2 \,dy\,dx
+\int_0^a\int_0^b y^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{x} &=& \int_0^a\int_0^b x(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b x \,dy\,dx
+\int_0^a\int_0^b x^3 \,dy\,dx
+\int_0^a\int_0^b xy^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{y} &=& \int_0^a\int_0^b y(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b y \,dy\,dx
+\int_0^a\int_0^b x^2y \,dy\,dx
+\int_0^a\int_0^b y^3 \,dy\,dx
\end{eqnarray}$$
Now you have nine integrals to solve, but they're all quite simple.
See the picture below.
As you can see, $\cos x$ is above $\sin x$ or $\cos x>\sin x$ for $\left[0,\dfrac\pi4\right]$. Therefore
$$
\begin{align}
M&=\int_0^\frac\pi4(\cos x-\sin x)\ dx=\sqrt{2}-1,\\\\
M_y&=\int_0^\frac\pi4x(\cos x-\sin x)\ dx=\frac14(\sqrt{2}\pi-4),\quad\Rightarrow\quad\text{use IBP}
\end{align}
$$
and
$$
\begin{align}
M_x&=\frac12\int_0^\frac\pi4(\cos x+\sin x)(\cos x-\sin x)\ dx\\
&=\frac12\int_0^\frac\pi4(\cos^2 x-\sin^2 x)\ dx\\
&=\frac12\int_0^\frac\pi4\cos2x\ dx\\
&=\frac14.
\end{align}
$$
I think you can handle it from here. I hope this helps.
Best Answer
Since formulas for $M_x$ and $M_y$ are $M_x =\displaystyle{\sum_{k=1}^{3}{m_ky_k}}$ and $M_y =\displaystyle{\sum_{k=1}^{3}{m_ky_k}}$, we have
$M_x=6(5)+5(-2)+10(-1)=30-10-10=10$
$M_y=6(1)+5(3)+10(-2)=6+15-20=1$
We set $M=m_1+m_2+m_3=6+5+10=21$
Then, the coordinates of the center of mass are $x=\dfrac{M_y}{M}=\dfrac{1}{21}$ and $y=\dfrac{M_x}{M}=\dfrac{20}{21}$.