Yes. The Poicare-Bendixon Theorem says that for a dynamical system defined in an open subset of $\mathbb R^2$, every compact, nonempty $\omega$ limit set of an orbit is either
- a periodic orbit
- a fixed point
- a countable number of homoclinic orbits connecting finitely many fixed points.
You found an open subset of $\mathbb R^2$ which is forward invariant with respect to your system, so the system is defined on an open subset of $\mathbb R^2$. This particular subset is important because it contains no fixed points. Therefore, the second and third options are out; you must have a periodic orbit. Since this set is bounded, every $\omega$ limit set is compact and nonempty.
These two versions of the Poincaré-Bendixson Theorem are not contradicting each other. Instead, I think you overlooked some of the hypotheses.
The first theorem tells you that a positively invariant, compact subset of the phase plane always contains at least one closed orbit, provided there are no fixed points in it (or provided it has just one unstable node or spiral point in its interior). While the proof of this fact relies on the topology of $\mathbb{R}^2$ a lot, the idea behind its formulation is easy to understand: if a trajectory is trapped in a compact subset of the phase plane which does not contain any stable fixed points, then it would have nothing to do but to approach a limit cycle.
The second theorem relies on different hypotheses: $R$ is still a positively invariant, compact subset of the phase plane, but it does not contain any periodic solutions now. So you should ask yourself: if a solution is forever trapped in $R$, and there are no limit cycles to be approached, what can that solution possibly do? Flowing toward a stable equilibrium point, of course. Indeed, this answers your question: it is possible to have a region $R$ that is a positively invariant, compact subset of the phase plane, with no periodic solutions in it, that still contains a finite number of nodes or spiral points: you'd just need some of those fixed points to be stable.
Also, pay attention to the fact your book says nothing about a positively invariant, compact subset of the phase plane that is both free from equilibrium points and limit cycles. Such a region cannot exist, and this is the very essence of the Poincaré-Bendixson Theorem: once you manage to trap a solution inside a region like $R$, then it can only either approach a (stable) fixed point, or spiral toward a limit cycle.
There's actually a third possibility, that of approaching a more complicated object called a cycle graph (which consists of a finite number of fixed points together with the orbits connecting them). Yet the spirit of the theorem does not change: a bounded solution may not wander in a chaotic fashion, because nothing different from approaching a stable fixed point, a limit cycle or a cycle graph is allowed. As a matter of fact, the Poincaré-Bendixson Theorem is what tells you that chaos can never occur in a two-dimensional dynamical system.
Best Answer
There is a small sign error in the trigonometric term in your solution for $\dot{r}$. A complete solution follows the sake future readers.
Problem statement
Is there a periodic solution for the following dynamical system?
$$ % \begin{align} % \dot{x} &= 4 x+2 y - x\left(x^2+y^2\right)\\ % \dot{y} &= -2 x+y-y \left(x^2+y^2\right) % \tag{1} \end{align} % $$
Solution method
Use the theorem of Poincare and Bendixson to identify a trapping region, here the gray annulus where the sign of the radial time derivative can change.
The invariant region must
Solution
Identify critical points
At what points $ \left[ \begin{array}{c} x \\ y \\ \end{array} \right] $ does $ \left[ \begin{array}{c} \dot{x} \\ \dot{y} \\ \end{array} \right] = \left[ \begin{array}{c} 0 \\ 0 \\ \end{array} \right] $? The only critical point is the origin.
Switch to polar coordinates
The workhorse formula is $$ % \begin{align} % x &= r \cos \theta, \\ % y &= r \sin \theta. % \end{align} % $$
With $r^{2} = x^{2} + y^{2}$, use implicit differentiation to find $$ r\dot{r} = x \dot{x} + y \dot{y} \tag{2} $$
Compute $\dot{r}$
Substituting into $(2)$ using $(1)$, and noting $\cos^{2} \theta = \frac{1}{2} \left( 1 + \cos 2\theta \right)$, $$ % \begin{align} % r \dot{r} &= x \dot{x} + y \dot{y} \\ % &= r^{2} - r^{4} \color{blue}{+} 3x^{2} \\ % &= r^{2} + \frac{3}{2} r^{2} \left( 1 \color{blue}{+} \cos 2\theta \right) - r^{4} % \end{align} % $$ Therefore $$ \dot{r} = -r^{3} + \frac{r}{2} \left( 5 \color{blue}{+} 3 \cos 2\theta \right) \tag{3} $$
Classify $\dot{r}$
Look for regions where the flow is outward $\dot{r}>0$, and regions where the flow is inward $\dot{r}<0$.
(Note the interesting comment by @Evgeny.)
Classify the problem by examining the limiting cases of $\cos 2\theta$ at $\pm 1$
Outward flow: $\cos 2 \theta \ge -1$
$$ % \begin{align} % \dot{r}_{in} &= -r^{3} + \frac{r}{2} \left( 5 + 3 (-1) \right) \\ % &= r(1-r^{2}) % \end{align} % $$ When $r<1$, $\dot{r}>0$, and the flow is outward.
Inward flow: $\cos 2 \theta \le 1$
$$ % \begin{align} % \dot{r}_{in} &= -r^{3} + \frac{r}{2} \left( 5 + 3 (-1) \right) \\ % &= r(4-r^{2}) % \end{align} % $$ When $r>2$, $\dot{r}<0$, and the flow is inward.
Trapping region
The region between the two zones is the annulus centered at the origin with inner and outer radii $$ % \begin{align} % r_{in} &= 1 \\ % r_{out} &= 2 % \end{align} % \tag{4} $$
There are no critical points. There region is closed and bounded. Therefore, a periodic solution exists.
Visualization
The vector field $\left[ \begin{array}{c} \dot{x} \\ \dot{y} \\ \end{array} \right]$ in $(1)$ is plotted against the gray trapping region in $(4)$. The red, dashed lines are nullclines which intercept at the critical point.