To integrate this region in polar coordinates,
it is advisable to break up the integral into two parts,
as shown in the figures below:
The two parts of the integral are divided by the diagonal line through
the upper right corner of the rectangle.
Since the sides of the rectangle are $a$ and $b$,
this diagonal line is at the angle $\arctan \frac ba.$
For $0 \leq \theta \leq \arctan \frac ba,$
you would integrate over $0 \leq r \leq a \sec\theta,$
and for $\arctan \frac ba \leq \theta \leq \frac\pi2,$
you would integrate over $0 \leq r \leq b \csc\theta.$
If you actually try this, I think you'll find that it is no easier than
doing the integration in rectangular coordinates.
It may even be worse.
An alternative approach, rather than combining $x^2+y^2$ into $r^2$,
is to integrate the terms separately:
$$\begin{eqnarray}
m &=& \int_0^a\int_0^b (1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b dy\,dx
+\int_0^a\int_0^b x^2 \,dy\,dx
+\int_0^a\int_0^b y^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{x} &=& \int_0^a\int_0^b x(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b x \,dy\,dx
+\int_0^a\int_0^b x^3 \,dy\,dx
+\int_0^a\int_0^b xy^2 \,dy\,dx
\end{eqnarray}$$
$$\begin{eqnarray}
m\bar{y} &=& \int_0^a\int_0^b y(1+x^2+y^2)\,dy\,dx \\
&=& \int_0^a\int_0^b y \,dy\,dx
+\int_0^a\int_0^b x^2y \,dy\,dx
+\int_0^a\int_0^b y^3 \,dy\,dx
\end{eqnarray}$$
Now you have nine integrals to solve, but they're all quite simple.
$$\bar{\cos{\theta}} = \int dr \, r^2 f(r) \, \int d\phi \, \sin{\phi} \, \int d\theta \, \cos{\theta}$$
$$\bar{\sin{\phi}} = \int dr \, r^3 f(r) \, \int d\phi \, \sin^2{\phi} \, \int d\theta \, $$
$$\bar{r} = \int dr \, r^3 f(r) \, \int d\phi \, \sin{\phi} \, \int d\theta \, $$
Then
$$\begin{align}\bar{r} \,\bar{\sin{\phi}}\,\bar{\cos{\theta}} &= \frac{\int dr \, r^3 f(r)\left (\int dr \, r^3 f(r) \right )^2 \int d\phi \, \sin^2{\phi} \left ( \int d\phi \, \sin{\phi} \right )^2\int d\theta \, \cos{\theta} \left ( \int d\theta \right )^2}{\left (\int dr \, r^2 f(r) \right )^3 \left ( \int d\phi \, \sin{\phi} \right )^3 \left ( \int d\theta \right )^3}\\ &= \frac{\int dr \, r^3 f(r) \, \int d\phi \, \sin^2{\phi} \,\, \int d\theta \, \cos{\theta} }{\int dr \, r^2 f(r)\, \int d\phi \, \sin{\phi}\, \int d\theta }\\ &= \bar{x}\end{align}$$
But I do not think you can work with the bare $\phi$ and $\theta$ because the trig functions introduce a nonlinearity that belies the linearity of the integrals.
Best Answer
The center of mass of a uniform half-disk obviously lies on the perpendicular bisector of the base diameter, at distance $d$ from the centre of the disk. By the Pappus centroid theorem,
$$ 2\pi d \cdot \frac{\pi}{2}R^2 = \frac{4\pi}{3}R^3, $$ hence $d=\color{red}{\large\frac{4R}{3\pi}}$.