To integrate this region in polar coordinates,
it is advisable to break up the integral into two parts,
as shown in the figures below:
The two parts of the integral are divided by the diagonal line through
the upper right corner of the rectangle.
Since the sides of the rectangle are $a$ and $b$,
this diagonal line is at the angle $\arctan \frac ba.$
For $0 \leq \theta \leq \arctan \frac ba,$
you would integrate over $0 \leq r \leq a \sec\theta,$
and for $\arctan \frac ba \leq \theta \leq \frac\pi2,$
you would integrate over $0 \leq r \leq b \csc\theta.$
If you actually try this, I think you'll find that it is no easier than
doing the integration in rectangular coordinates.
It may even be worse.
An alternative approach, rather than combining $x^2+y^2$ into $r^2$,
is to integrate the terms separately:
$$\begin{eqnarray}
m &=& \int_0^a\int_0^b (1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b dy\,dx
+\int_0^a\int_0^b x^2 \,dy\,dx
+\int_0^a\int_0^b y^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{x} &=& \int_0^a\int_0^b x(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b x \,dy\,dx
+\int_0^a\int_0^b x^3 \,dy\,dx
+\int_0^a\int_0^b xy^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{y} &=& \int_0^a\int_0^b y(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b y \,dy\,dx
+\int_0^a\int_0^b x^2y \,dy\,dx
+\int_0^a\int_0^b y^3 \,dy\,dx
\end{eqnarray}$$
Now you have nine integrals to solve, but they're all quite simple.
Actually, the factor of $2$ is unimportant. The center of mass in the $y$ direction is given by
$$\bar{y} = \frac{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)(r \sin{\theta})}{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)}$$
Note that the weight of the $y$ coordinate is expressed in the $r \sin{\theta}$ term in the numerator. Note also the boundary of the lamina is expressed in its polar form, $r=\sin{\theta}$. Evaluating the radial integrals, we get
$$\bar{y} = \frac{\frac{16}{4} \displaystyle \int_0^{\pi/2} d\theta \: \sin^5{\theta}}{\frac{8}{3} \displaystyle \int_0^{\pi/2} d\theta \: \sin^3{\theta}}$$
or
$$\bar{y} = \frac{6}{5}$$
For $\bar{x}$, we do a similar calculation:
$$\begin{align}\bar{x} &= \frac{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)(r \cos{\theta})}{\displaystyle \int_0^{\pi/2} d\theta \: \int_0^{2 \sin{\theta}} dr \, r (2 r)}\\ &= \frac{\frac{16}{4} \displaystyle \int_0^{\pi/2} d\theta \: \cos{\theta} \sin^4{\theta}}{\frac{8}{3} \displaystyle \int_0^{\pi/2} d\theta \: \sin^3{\theta}}\\ &= \frac{9}{20}\end{align}$$
Best Answer
The mass of a rectangular lamina with boundaries $ \ a \le x \le b \ $ and $ \ c \le y \le d \ $ is given by
$$ M \ = \ \int_c^d \ \int_a^b \ \ \rho (x,y) \ \ dx \ dy \ . $$
The coordinates of the centroid ("center-of-mass" in physical applications) are found from
$$ \overline{x} \ = \ \frac{\int_c^d \ \int_a^b \ \ x \cdot \rho (x,y) \ \ dx \ dy}{M} \ \ , $$
$$ \overline{y} \ = \ \frac{\int_c^d \ \int_a^b \ \ y \cdot \rho (x,y) \ \ dx \ dy}{M} \ \ . $$
Note that since the density gets larger as we go toward the "upper-right" corner, $ \ (7,2) \ , $ we should expect the centroid to be "above and to the right" of the center of the rectangle at $ \ ( \frac{7}{2} , 1 ) \ . $ [Later, I'll mention a handy way to get the mass without integration when the density function is linear in each dimension...]
For the sake of guidance, I get $ \ M \ = \ 273 \ , \ \overline{x} \ = \ \frac{1127}{273} \ , \ \overline{y} \ = \ \frac{875}{3 \ \cdot \ 273} \ . $ (No one said the centroid coordinates would be pretty -- just rational...)
EDIT: The "short-cut" I was holding off mentioning is that when linear functions are involved, some calculations can be greatly simplified because of the way some integrals work. The density function in this problem, $ \ \rho(x, y) = 3x + 4y + 5 \ $ , is linear in both the $ \ x-$ and $ \ y-$ directions. In this situation, the mass of the lamina is just the density at the geometrical center of the region (not the centroid; the two only coincide for uniform density) times the area. For this problem, we thus find
$$ M \ = \ A \ \cdot \ \rho(x_m, y_m) \ = \ (b-a) \ (d-c) \cdot \ \rho(\frac{a+b}{2},\frac{c+d}{2}) $$
$$ = \ 7 \ \cdot \ 2 \ \cdot \ \rho(\frac{7}{2},1) \ = \ 14 \ \cdot (3 \cdot \frac{7}{2} \ + \ 4 \cdot 1 \ + \ 5) \ = \ 14 \ \cdot \ 19.5 \ = \ 273\ \ . $$