You don't need to use Polar Coordinates.
With the same set-up as you have described, consider semi-circular elements of radius $x$ and thickness $\delta x$
Then the area element is $\delta A=\pi x\delta x$
Now if the density per unit area is $\rho =kx$, then the mass element is $$\delta m=\pi kx^2\delta x$$
Integrating gives the mass of the lamina, as you have found, is $$m=\frac 13 \pi kr^3$$
Now using the formula for the centroid of an arc, the distance of the centroid of the semicircular element from the origin is $$\frac {x\sin \left(\frac {\pi}{2}\right)}{\frac {\pi}{2}}=\frac {2x}{\pi}$$
Now applying Varignon's Principle, $$\frac 13 \pi kr^3\bar x=\int_0^r\frac{2x}{\pi}\pi kx^2dx=\frac{kr^4}{2}$$
Hence the result $$\bar x=\frac{3r}{2\pi}$$
I am doing this in two stages. The disc needs to be divided into elements in the form of infinitesimally thin and concentric hoops, which we then integrate. But first we have to find the MI of a hoop about an axis at angle $\frac {\pi}{6}$ to the plane of the hoop, through the centre of the hoop.
Consider a circle of radius $r$ in the $xy$ plane, centre $O$, and a point on the circumference with coordinates $P(r\cos\theta,r\sin\theta, 0)$.
We will write the axis, as specified in the question, as the line $$\underline{r}=t\left(\begin{matrix}\frac{\sqrt{3}}{2}\\0\\\frac 12\end{matrix}\right)$$
The distance $d$ from $P$ to the line is given by the magnitude of the cross product $$\overrightarrow{OP}\times\underline{\hat{r}}$$
A quick calculation gives $$d^2=\frac 14r^2(1+3\sin^2\theta)$$
Now let the circle be a hoop of mass $m$ and density per unit length $\rho$
The MI of an element of mass $\delta m$ at $P$ is $\delta I=\delta m d^2=\rho r \delta\theta d^2$
Therefore for the whole hoop, $$I=\frac 14\rho r^3\int_0^{2\pi}(1+3\sin^2\theta)d\theta=\frac 58mr^2$$
Now finally we can consider the disc of radius$R$, mass $m$ and density per unit are $\rho$ divided into concentric elements of thickness $\delta x$ and radius $x$
Then $$\delta I=\frac 58\delta mx^2=\frac 54\pi\rho x^3\delta x$$
$$\Rightarrow I=\frac 54\pi\rho\int_0^Rx^3dx$$
$$\Rightarrow I=\frac{5}{16}mR^2$$
Best Answer
We should assume that this shell $S$ is infinitely thin, but has mass $1$ per unit of area.
An "infinitesimal latitude zone" of width $ds$ at geographical latitude $\theta\in\bigl[0,{\pi\over2}\bigr]$ has area $$d\omega=2\pi R \cos\theta\>ds=2\pi R^2 \cos\theta\>d\theta$$ and height $$z=R\sin\theta$$ over ground. The $z$-coordinate of the center of mass of $S$ is therefore given by $$z_c={\int_S z\>d\omega\over\int_S d\omega}={\pi R^3\int_0^{\pi/2}2\cos\theta\sin\theta\>d\theta \over2\pi R^2}={R\over2}\ .$$ If the hemispherical shell $H$ has a positive thickness, i.e., an inner radius $a$ and an outer radius $b$, then one has to compute volume integrals. The coordinates to choose are then spherical, and not cylindrical, coordinates. The volume element is given by $${\rm d}V=r^2\cos\theta\>{\rm d}(r,\phi,\theta)\ .$$ Here, as before, $\theta=0$ at the equator, and $r^2\cos\theta$ is the Jacobian. Since we still have $z=r\sin\theta$ we now obtain $$z_c={\int_H z\>{\rm d}V\over \int_H {\rm d}V}\ .\tag{1}$$ By elementary geometry one has $$ \int_H {\rm d}V={2\pi\over3}(b^3-a^3)\ ,$$ and the numerator in $(1)$ can be "separated" into $$\int_H z\>{\rm d}V=2\pi \int_a^b r^3\>dr\cdot \int_0^{\pi/2}\sin\theta\cos\theta\>d\theta$$ (note that the integration with respect to $\phi$ is for free). I may leave the last two integrals to you.