There was lemniscate trouble, as you feared. Up to $\theta=\frac{\pi}{4}$, everything was fine. But from $\frac{\pi}{4}$ and for quite a while (up to $\frac{3\pi}{4}$), $\cos 2\theta$ is negative, so $r^2$ is negative and there is no curve. The integral doesn't know and doesn't care: it cheerfully "adds up" these negatives, giving the wrong answer.
So for the limaçon part, if you do things in your style, you will have to break up the limaçon integral into two integrals, $\frac{\pi}{6}$ to $\frac{\pi}{4}$ and then $\frac{3\pi}{4}$ to $\frac{5\pi}{6}$.
Since the integrals for the lemniscate and the limaçon are over different intervals, we cannot express their difference as a single integral.
What I would do is to use the symmetry, take the right-half of the region and multiply the resulting area by $2$. The limaçon part to be subtracted uses the integral from $\frac{\pi}{6}$ to $\frac{\pi}{4}$.
So there is $\frac{1}{2}$ as in your formula, but ultimately we multiply by $2$, so our area is
$$\int_{\pi/6}^{\pi/2}(3+2\sin \theta)^2\,d\theta-\int_{\pi/6}^{\pi/4}32\cos 2\theta\,d\theta.$$
More pleasant, fewer minus signs to worry about!
Best Answer
Make a careful sketch. Or have software do it for you. We want the area that is common to the regions enclosed by the two curves.
The two curves meet at $\theta=\pi/6$ and $\theta=\pi-\pi/6$.
Looking outward from the origin, from $\theta=0$ to $\theta=\pi/6$, the first curve we meet is the circle. Then from $\pi/6$ to $\pi/2$ (and beyond), it's the other guy up to $5\pi/6$. Then it is the circle again up to $\pi$. But there is symmetry, so go to $\pi/2$ only and double.
For each part integrate $\frac{1}{2}r^2\,d\theta$. Since we intended to double, forget about the $\frac{1}{2}$ and forget about doubling. We get $$\int_0^{\pi/6} 9\sin^2\theta\,d\theta + \int_{\pi/6}^{\pi/2}(2-\sin\theta)^2\,d\theta.$$