We show how to do the area calculation using polar coordinates.
There is a potential minor complication. There are two different conventions about the meaning of a polar equation when $r<0$. Some say the curve is then not defined. Some say it is defined, but one must reflect across the origin. For our particular curve, the two conventions give the same region, so we don't need to worry. But for other curves, you may need to know what convention your course uses.
The curve $r=6\sin \theta$ and the circle $r=1$ meet where $\sin\theta=1/6$. Let $\theta_0=\arcsin(1/6)$. From $\theta=\theta_0$ to $\theta=\pi-\theta_0$, the curve $r=6\sin\theta$ is "outside" the circle $r=1$. By symmetry, we can just look at the part from $\theta=\theta_0$ to $\theta=\pi/2$, and double the resulting area.
The area from $\theta=\theta_0$ to $\theta=\pi/2$, bounded by our curve $r=6\sin\theta$, is
$$\int_{\theta_0}^{\pi/2} (1/2)r^2 d\theta.$$
From this we must subtract the area inside the circle $r=1$, which is
$$\int_{\theta_0}^{\pi/2} (1/2)\,1^2 \,d\theta.$$
Subtract, and multiply by $2$ to take care of the other half of the region.
The desired area is therefore
$$\int_{\theta_0}^{\pi/2} (36\sin^2\theta-1)\, d\theta.$$
Now integrate. Using the identity $\cos 2\theta=1-2\sin^2\theta$, we find that $36\sin^2\theta=18-18\cos 2\theta$. Thus our area is
$$\int_{\theta_0}^{\pi/2} (17-18\cos 2\theta)\, d\theta.$$
The rest is routine. We will need $\sin 2\theta_0$. When you calculate this, you will bump into the $35$ that you already met in your rectangular coordinates analysis. The ultimate answer is not "nice," because $\arcsin(1/6)$ is not a pleasant number. Too bad that the polar curve was not $r=2\sin\theta\:$!
Comment: We can also obtain the result by switching to rectangular coordinates, as you did. The switching in this case was straightforward, and showed we were dealing with a circle. However, in many cases the rectangular coordinates version of a curve given in polar coordinates can be very unpleasant. It is therefore useful to learn to work directly in polar coordinates.
In situations where there is strong circular symmetry, polar coordinates come up very naturally. For example, the force of gravity exerted by the Earth on a small object is directed towards the center of the Earth. In studying motions of satellites, polar coordinates are the way to go.
Best Answer
Since you already have an idea what the functions look like, here's a pretty diagram.
Now obviously, our area is equal to
$$\frac\pi2a^2 + 2\int_{0}^{\pi/2}\frac12[a(1-\sin(\theta))]^2\,d\theta$$
Think you can take it from here?