The region you are integrating over is the part of $y=3-x^2$ that lies above the $x$-axis. If you are going to describe it using polar coordinates, then $\theta$ should only range from $\theta=0$ (to consider the line from $(0,0)$ to $(\sqrt{3},0)$) to $\theta=\pi$ (which gives you the line from $(0,0)$ to $(-\sqrt{3},0)$.
So $\theta$ will only range from $0$ to $\pi$, not from $0$ to $2\pi$. What about $r$?
You want to express $r$ as a function of $\theta$. Your graph is $y=3-x^2$ (you seem to have forgotten the $y$...) So the curve you are trying to express would correspond to:
$$\begin{align*}
y &= 3-x^2\\
r\sin\theta &= 3-r^2\cos^2\theta
\end{align*}$$
So you want to express $r$ as a function of $\theta$, so that you can have that in the "inner" integral (the limits of integration for $r$ may depend on $\theta$, but they should not depend on $r$). So we need to solve for $r$; this gives you a quadratic in $r$:
$$(\cos^2\theta) r^2 + (\sin\theta)r - 3 = 0.$$
Thus, the graph corresponds to
$$r = \frac{-\sin\theta \pm \sqrt{\sin^2\theta +12\cos^2\theta}}{2\cos^2\theta} = \frac{-\sin\theta \pm \sqrt{1 + 11\cos^2\theta}}{2\cos^2\theta}.$$
Since $r$ is positive for the region you want, you would use the $+$ sign. So the region you want would correspond to
$$\begin{align*}
0 &\leq r \leq \frac{-\sin\theta + \sqrt{1+11\cos^2\theta}}{2\cos^2\theta}\\
0 &\leq \theta \leq \pi.
\end{align*}$$
And your integral would be
$$\int_0^{\pi}\int_0^{\frac{-\sin\theta + \sqrt{1+11\cos^2\theta}}{2\cos^2\theta}} 1r\,dr\,d\theta$$
... at which point I would grimace in disgust and switch back to cartesian coordinates, because this is definitely not a nice way to go...
(The issue, of course, is that while circles can be described nicely with polar coordinates, parabolas in general are somewhat nasty.)
There are already a lot of good answers here, so I'm adding this one
primarily to dazzle people w/ my Mathematica diagram-creating skills.
As noted previously,
$x(t)=a \cos (t)$
$y(t)=b \sin (t)$
does parametrize an ellipse, but t is not the central angle. What
is the relation between t and the central angle?:
Since y is bSin[t] and x is aCos[t], we have:
$\tan (\theta )=\frac{b \sin (t)}{a \cos (t)}$
or
$\tan (\theta )=\frac{b \tan (t)}{a}$
Solving for t, we have:
$t(\theta )=\tan ^{-1}\left(\frac{a \tan (\theta )}{b}\right)$
We now reparametrize using theta:
$x(\theta )=a \cos (t(\theta ))$
$y(\theta )=b \sin (t(\theta ))$
which ultimately simplifies to:
$x(\theta)=\frac{a}{\sqrt{\frac{a^2 \tan ^2(\theta )}{b^2}+1}}$
$y(\theta)=\frac{a \tan (\theta )}{\sqrt{\frac{a^2 \tan ^2(\theta )}{b^2}+1}}$
Note that, under the new parametrization, $y(\theta)/x(\theta) =
tan(\theta)$ as desired.
To compute area, we need $r^2$ which is $x^2+y^2$, or:
$r(\theta )^2 = (\frac{a}{\sqrt{\frac{a^2 \tan ^2(\theta )}{b^2}+1}})^2+
(\frac{a \tan (\theta )}{\sqrt{\frac{a^2 \tan ^2(\theta )}{b^2}+1}})^2$
(note that we could take the square root to get r, but we don't really need it)
The above ultimately simplifies to:
$r(\theta)^2 =
\frac{1}{\frac{\cos ^2(\theta )}{a^2}+\frac{\sin ^2(\theta )}{b^2}}$
Now, we can integrate $r^2/2$ to find the area:
$A(\theta) = (\int_0^\theta
\frac{1}{\frac{\cos ^2(x )}{a^2}+\frac{\sin ^2(x )}{b^2}} \, dx)/2$
which yields:
$A(\theta) = \frac{1}{2} a b \tan ^{-1}\left(\frac{a \tan (\theta )}{b}\right)$
good for $0\leq \theta <\frac{\pi }{2}$
Interestingly, it doesn't work for $\theta =\frac{\pi }{2}$ so we
can't test the obvious case without using a limit:
$\lim_{\theta \to \frac{\pi }{2}} \, \frac{1}{2} a b \tan
^{-1}\left(\frac{a \tan (\theta )}{b}\right)$
which gives us $a*b*Pi/4$ as expected.
Best Answer
Your first answer is twice the correct answer for the following reason: if you let $\theta$ range from $\theta=0$ to $\theta=2\pi$, the curve $r=4\cos(3\theta)$ — which is a flower with three petals — is traced twice, and therefore you find twice the area. If you trace it carefully starting from $\theta=0$, which is $(4,0)$ in cartesian coordinates, you will see that the curve is completed and comes back to the initial point at $\theta=\pi$; and then, from $\theta=\pi$ to $\theta=2\pi$ you retrace it once more.
In the second method, you find the area of a half of one petal, which you correctly determined to range from $\theta=0$ to $\theta=\dfrac{\pi}{6}$. Since there are six such half-petals, multiplying by $6$ clearly yields the correct answer. Note, however, that taking six half-petals of the same "angular width" (so to speak) as the one going from $\theta=0$ to $\theta=\dfrac{\pi}{6}$ will produce the angle six times as wide, i.e. from $\theta=0$ to $\theta=\pi$, consistent with my explanation above.