It is not quite clear how Poincaré-Bendixson theorem could prove useful in this setting since the unit circle acts as a closed orbit of this dynamical system, see the picture below. Perusing these (reasonably concise) lecture notes on the subject might prove useful.
To solve the question, consider any orbit starting from some initial condition $(r_0,\theta_0)$ with $r_0\gt0$. The phase diagram of $\dot r=r(1-r)$ on $r\gt0$ shows that $r(t)\to1$, in particular $1\leqslant2r(t)\leqslant4$ for every $t$ large enough, say, for every $t\geqslant t_1$.
On $t\geqslant t_1$, $\dot\theta(t)=2r(t)\sin^2(\theta(t)/2)$ hence $\sin^2(\theta/2)\leqslant\dot\theta\leqslant4\sin^2(\theta/2)$. This implies that $t\mapsto\theta(t)$ follows an orbit of the dynamical system $$\dot\varphi=\sin^2(\varphi/2),$$ and the relative speed of $\theta$ compared to $\varphi$ is always between $1$ and $4$, in particular $\theta(t)$ converges to the first fixed point met by $\varphi(t)$, which, for every starting condition $\varphi_0=\theta_0$, is $0$ mod $2\pi$.
This proves that, for every $r_0\gt0$, $r(t)\to1$ and $\theta(t)\to0$ mod $2\pi$, that is, for every initial condition $(x_0,y_0)\ne(0,0)$, the orbit $(x(t),y(t))$ accumulates on $(1,0)$.
The simplest proof that the $(x,y)$-point $(1,0)$ is unstable might be to consider an initial condition on the unit cercle $r_0=1$ with $\theta_0$ in $(0,2\pi)$. Then, for every $t$, $$r(t)=1,\qquad \dot\theta(t)=2\sin^2(\theta(t)/2),$$ hence $\theta(t)$ crosses all the interval $[\theta_0,2\pi)$, that is, $(x(t),y(t))$ moves on the unit circle anti-clockwise from the angle $\theta_0$ to the angle $2\pi$. In particular, when $\theta_0\to0$, $\theta_0\gt0$, the trajectory $(x(t),y(t))$ always passes by $(x,y)=(-1,0)$, which is not at vanishing distance from $(0,0)$. This proves that $(1,0)$ is unstable.
$\qquad\qquad\qquad\qquad$
streamplot[{x(1+y)−(x+y)sqrt(x^2+y^2),y-x^2+(x-y)sqrt(x^2+y^2)},{x,-2,2},{y,-2,2}]
Best Answer
Consider the Lyapunov function $V(x,y) = \frac{1}{2}(x^2+y^2)$. For a Lyapunov function $V$, $\dot{V}<0$ for $(x,y)\neq0$ implies that the dynamical system is asymptotically stable. In this case, we have $$ \dot{V} = x\dot{x}+y\dot{y} = -x^2+xy-y^2-y^4. $$ You can use basic calculus to show that $\dot{V}(x,y)$ has a global maximum at $(x,y)=0$, (take the gradient and set equal to 0). We also have $\dot{V}(0,0) = 0$, so $\dot{V}<0$ for $(x,y)\neq0$ and the system is asymptotically stable