There isn't much difference between doing area integration in polar coordinates as a double integral and in the way you may have encountered it earlier in single-variable calculus. It is still important to have an idea of what the regions look like (here, you have a limacon and a "peanut").
Your radial portion is correct, but you really need to look at the angles where the curves intersect: you'll want to solve $ \ 2 + \sin \theta \ = \ 2 + \cos (2 \theta)$ to get the range of angle integration. There are two zones to cover, but you can make use of symmetry here and just integrate over one of them.
The red curve is the limacon $2 + \sin\theta$ , the blue curve, $2 + \cos(2 \theta)$ .
Incidentally, in the integral you wrote, it's not that there is no integrand, but rather that the integrand is "1". You are doing a surface integral where all infinitesimal patches are equally "weighted" at 1 , so the result is simply the area of the region. Surface integrals with some "weighting function" $f(r,\theta)$ or $g(x,y)$ turn up in various applications.
This polar area integration does have a couple of bits that require careful handling, in part because of the character of the two curves, and partly because one of the limits of integration is not a "fundamental angle".
The curve $ \ r \ = \ 2 \cos \theta \ $ is a "one-petal" rosette, the sort that looks like a circle tangent to the origin with its center on the $ \ x-$ axis. What is special about rosettes with an odd number of petals is that the curves are completely "swept out" with a period of $ \ \pi \ $ , rather than $ \ 2 \pi \ . $ The other curve, $ \ r^2 \ = \ 16 \ \cos (2\theta) \ , $ is a lemniscate, which has the peculiar property that its polar equation produces non-real radii for half of its period .
We wish to find the area of the region within both curves (shown in green). Since it is symmetrical about the $ \ x-$ axis, we will just integrate over the half above that axis and double the result. We need to know how the angle-variable "runs" along each curve, and the value of $ \ \theta \ $ at which the two curves intersect.
Both curves intercept the $ \ x-$ axis at $ \ \theta = 0 \ , $ but each first reaches the origin at distinct values, the rosette at $ \ \theta = \frac{\pi}{2} \ , $ the lemniscate at $ \ \theta = \frac{\pi}{4} \ . $ Since we will be working in the first quadrant, it is "safe" to solve for the intersection point of the curves by equating $ \ r^2 \ $ for the two: this will not introduce "spurious" solutions. We obtain
$$ 16 \ \cos 2 \theta \ = \ 4 \ \cos^2 \theta \ \ \Rightarrow \ \ 4 \ \cos 2 \theta \ = \ \frac{1}{2} ( \ 1 \ + \ \cos 2\theta \ ) $$
$$ \Rightarrow \ \ 8 \ \cos 2 \theta \ = \ 1 \ + \ \cos 2\theta \ \ \ \Rightarrow \ \ \cos 2 \theta \ = \ \frac{1}{7} \ \ , $$
employing familiar trigonometric identities. The solution for $ \ \theta \ $ is not any "fundamental angle" (it proves to be $ \ \Theta \ \approx \ 0.7137 \ $ , as a graph of the functions will verify), so we will actually prefer to work with our result for $ \ \cos 2 \Theta \ . $ (We will shortly have need of the value $ \ \sin 2 \Theta \ = \ \frac{\sqrt{48}}{7} \ . $ )
The "upper half" of the area of the region is covered then by integrating the area within the rosette from $ \ \theta = 0 \ \ \text{to} \ \ \theta = \Theta \ , $ and then passing over to the lemniscate from $ \ \theta = \Theta \ \ \text{to} \ \ \theta = \frac{\pi}{4} \ . $ (This does behave properly, since $ \ \Theta \ < \ \frac{\pi}{4} \ \approx \ 0.7854 \ . $ ) The area of the entire region is then found from
$$ 2 \ \left[ \ \int_0^{\Theta} \ \frac{1}{2} r^2_{ros} \ \ d\theta \ \ + \ \ \int^{\pi / 4}_{\Theta} \ \frac{1}{2} r^2_{lemn} \ \ d\theta \ \right] $$
$$ = \ \ \int_0^{\Theta} \ ( \ 2 \cos \theta \ )^2 \ \ d\theta \ \ + \ \ \int^{\pi / 4}_{\Theta} ( \ 16 \ \cos 2\theta \ ) \ \ d\theta $$
$$ = \ \ 4 \ \int_0^{\Theta} \cos^2 \theta \ \ d\theta \ \ + \ \ 16 \ \int^{\pi / 4}_{\Theta} \cos 2\theta \ \ d\theta $$
$$ = \ \ 2 \ \int_0^{\Theta} ( \ 1 + \cos 2 \theta \ ) \ \ d\theta \ \ + \ \ 16 \ \int^{\pi / 4}_{\Theta} \cos 2\theta \ \ d\theta $$
$$ = \ \ \left( \ 2 \theta \ + \ \sin 2 \theta \ \right) \vert_0^{\Theta} \ + \ \left( \ 8 \ \sin 2 \theta \ \right) \vert^{\pi / 4}_{\Theta} $$
$$ = \ \ ( \ 2 \Theta \ + \ \sin 2 \Theta \ - \ 0 \ - \ 0) \ + \ ( \ 8 \ - \ 8 \ \sin 2 \Theta \ ) $$
$$ = \ \ 8 \ + \ 2 \Theta \ - \ 7 \ \sin 2 \Theta \ = \ 8 \ + \ \arccos \frac{1}{7} \ - \ 7 \ \cdot \frac{\sqrt{48}}{7} $$
$$ = \ 8 \ + \ \arccos \frac{1}{7} \ - \ \sqrt{48} \ \approx \ 2.4992 \ \ . $$
The region lies within the rosette, which has area $ \ \pi \ , $ so this result is reasonable. (In fact, it fills very close to 80% of the rosette.)
Best Answer
Side Question
There are lots of different contexts/formalizations with involved manipulation of differentials. Trying to give an overview of all of them would be a big undertaking worthy of a whole separate question. Very briefly: The most common thing that is taught/that I would expect a mathematician to think of is the theory of "differential forms". But there are also many other relevant things like "geometric calculus", "Smooth Infinitesimal Analysis", Bartlett and Khurshudyan's approach to manipulating higher order differentials (arXiv link), debatably: Robinson's approach to nonstandard analysis, and probably a couple other things I'm forgetting or haven't encountered.
Main Question
Disclaimer
This is almost entirely copied from my answer to the similar question Why can't we convert the area element $dA$ to polar by multiplying the polar expressions for $dx$ and $dy$?.
Intro
There are two main types of ways to think about things like $\mathrm{d}x\mathrm{d}y$ in multivariable calculus, and we often switch between them depending on the context. (This clarification was inspired in part by Terry Tao's preprint on “differential forms and integration”.) $\mathrm{d}x$ can either act kind of like a number, or act kind of like a vector.
For the “number” interpretation, there are things like limit arguments or infinitesimals in nonstandard analysis to make things rigorous. For the “vector” interpretation, there are things like “differential forms” or “geometric calculus” to make things rigorous. But I'm going to gloss over those details because there are many ways to make things formal, and the exact choices don't affect the intuition here.
Numbers
One way to think about things is that $\mathrm{d}x$ and $\mathrm{d}y$ are in some way like tiny positive numbers representing the width and length of a tiny rectangle, so that $\mathrm{d}x\mathrm{d}y$ is the area of a tiny rectangle. Then when we write something like $\iint f\left(x,y\right)\,\mathrm{d}x\mathrm{d}y$ or $\iint g\left(r,\theta\right)\,\mathrm{d}r\mathrm{d}\theta$, we just add up the signed volumes (in case $f$ or $g$ is negative) of thin rectangular prisms with cross-sectional area represented by $\mathrm{d}x\mathrm{d}y$ or $\mathrm{d}r\mathrm{d}\theta$.
Under this interpretation, $\mathrm{d}x=\mathrm d(r\cos\theta)=\cos\theta\mathrm{d}r-r\sin\theta\mathrm{d}\theta$ doesn't make too much sense. For example, if $\theta=\pi/2$, then we would have $\mathrm{d}x=-r\mathrm{d}\theta$, so that $\mathrm{d}\theta$ and $\mathrm{d}x$ couldn't both represent positive lengths. But we can still understand the relationship between the areas $\mathrm{d}x\mathrm{d}y$ and $\mathrm{d}r\mathrm{d}\theta$ with arguments like the geometric one in this answer by Mike Spivey.
Vectors
The other way to think about things is that $\mathrm{d}x$ and $\mathrm{d}y$ are in some way like tiny vectors whose direction we care about, and this leads to a slightly different discussion. To emphasize this vector idea, I will use some nonstandard notation. Let's write $\overrightarrow{\mathrm{d}x}=\left\langle \Delta x,0,0\right\rangle$ for some positive $\Delta x$, and $\overrightarrow{\mathrm{d}y}=\left\langle 0,\Delta y,0\right\rangle$ for some positive $\Delta y$. So $\overrightarrow{\mathrm{d}x}$ points to the right in the $xy$-plane and $\overrightarrow{\mathrm{d}y}$ points “up” in the $xy$-plane. Then the area of the little rectangle they make is $\left\Vert \overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}\right\Vert =\Delta x\Delta y$.
However, now that we have vectors, we could choose to care about the orientation. When we think about a usual integral like $\int_{\left[a,b\right]}f\left(x\right)\,\mathrm{d}x$ when $f$ is negative, we decide to count that area in a negative way. Similarly, we could count the “signed area” of the oriented rectangle with sides $\overrightarrow{\mathrm{d}x},\overrightarrow{\mathrm{d}y}$ as the positive value $\Delta x\Delta y$ (which is just the $z$-component of $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$). But the signed area of the oriented rectangle with sides $\overrightarrow{\mathrm{d}y},\overrightarrow{\mathrm{d}x}$ would be the negative value $-\Delta x\Delta y$ (which is just the $z$-component of $\overrightarrow{\mathrm{d}y}\times\overrightarrow{\mathrm{d}x}$).
Then when we write something like $\iint f\left(x,y\right)\,\mathrm{d}x\mathrm{d}y$ or $\iint g\left(r,\theta\right)\,\mathrm{d}r\mathrm{d}\theta$, we could (if we chose) care about the order of things, with something like an implied $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$ or $\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}$ in our heads, if not always our notation.
When using things like vectors, then something like $\overrightarrow{\mathrm{d}x}=\cos\theta\overrightarrow{\mathrm{d}r}-r\sin\theta\overrightarrow{\mathrm{d}\theta}$ makes a lot of sense. $\overrightarrow{\mathrm{d}r}$ points away from the origin, and $\overrightarrow{\mathrm{d}\theta}$ points perpendicularly to $\overrightarrow{\mathrm{d}r}$ in the counterclockwise way (so that $\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}$ points in the same positive-$z$ direction as $\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}$).
Finally, we can correct your calculation:
$$\overrightarrow{\mathrm{d}x}\times\overrightarrow{\mathrm{d}y}=\left(\cos\theta\overrightarrow{\mathrm{d}r}-r\sin\theta\overrightarrow{\mathrm{d}\theta}\right)\times\left(\sin\theta\overrightarrow{\mathrm{d}r}+r\cos\theta\overrightarrow{\mathrm{d}\theta}\right)$$ $$=\cos\theta\sin\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}r}-r^{2}\sin\theta\cos\theta\overrightarrow{\mathrm{d}\theta}\times\overrightarrow{\mathrm{d}\theta}+r\cos^{2}\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}-r\sin^{2}\theta\overrightarrow{\mathrm{d}\theta}\times\overrightarrow{\mathrm{d}r}$$ $$=\overrightarrow{0}-\overrightarrow{0}+r\cos^{2}\theta\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}-r\sin^{2}\theta\left(-\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}\right)=\boxed{r\,\overrightarrow{\mathrm{d}r}\times\overrightarrow{\mathrm{d}\theta}}$$