We are given:
$$\dot x=-y(x^2+y^2), \dot y=x(x^2+y^2)$$
Recall, when we are doing polar coordinates, we have
$$x = r \cos \theta, y = r \sin \theta, x^2+y^2 = r^2$$
When we differentiate this, we have:
$$2 x x' + 2 y y' = 2 r r'$$
This gives us:
$$rr' = x(-y(x^2+y^2)) + y(x(x^2+y^2)) = 0 \rightarrow r' = 0$$
To find the angle, we take:
$$\dfrac{r \sin \theta}{r \cos \theta} = \tan \theta = \dfrac{y}{x}$$
Using the quotient rule gives us:
$$\theta' = \dfrac{x y' - y x'}{r^2} = \dfrac{x^2(x^2+y^2)+y^2(x^2+y^2)}{r^2}= \dfrac{r^4}{r^2} = r^2$$
Thus in polar coordinates, the original system becomes
Can you now solve this system and draw the phase portrait?
The phase portrait should look like:
From the equations $x= r\cos\theta$ and $y = r\sin \theta$ we see that the derivatives of $r$ and $\theta$ are related to the derivatives of $x$ and $y$ by the equations
\begin{eqnarray*}
\dot x & = & \dot r \cos \theta - \dot \theta r\sin \theta\\
\dot y & = & \dot r \sin \theta + \dot \theta r\cos \theta.
\end{eqnarray*}
Thus, using the notation $g_i(r, \theta) = f_i(r\cos \theta, r\sin\theta)$ (i = 1, 2), the given system of ODEs reads
\begin{eqnarray*}
\dot r \cos \theta - \dot \theta r\sin \theta & = & g_1(r, \theta) \\
\dot r \sin \theta + \dot \theta r\cos\theta & = & g_2(r, \theta) .
\end{eqnarray*}
Solving this for $\dot r $ and $\dot \theta$ gives
\begin{eqnarray*}
\dot r & = & g_1(r, \theta)\cos \theta + g_2(r, \theta) \sin \theta\\
\dot \theta & = & \frac{g_2(r, \theta)}{r}\cos\theta - \frac{g_1(r, \theta)}{r} \sin \theta.
\end{eqnarray*}
Evidently, the right-hand side of the second equation has a problem at $r = 0$.
To see why the problem at $r = 0$ is harmless when $(x,y) = (0,0)$ is a rest point, consider for $i = 1, 2$ the Taylor expansion of $g_i$ about $(r,\theta) = (0,0)$. We have
\begin{eqnarray*}
g_i(r, \theta)
& = & g_i(0,0) + \partial_rg_i(0,0)r + \partial_\theta g_i(0,0)\theta + {\rm HOT’s}\\
& = & f_i(0,0) + \left(\partial_x f_i(0,0) \cos\theta + \partial_y f_i(0,0) \sin\theta\right)r + \left(- \partial_x f_i(0,0) r\sin \theta + \partial_y f_i(0,0) r\cos\theta\right)\theta + {\rm HOT’s}\\
& = &
f_i (0,0) + r\big[\partial_x f_i(0,0) \cos\theta + \partial_y f_i(0,0) \sin\theta + \left(- \partial_x f_i(0,0)\sin \theta + \partial_y f_i(0,0)\cos\theta\right)\theta\big] + {\rm HOT’s}
\end{eqnarray*}
From this formula, we see that if $f_i(0,0) = 0$ then $g_i$ is of the form
\begin{equation*}
g_i(r, \theta) = rh_i(r, \theta)
\end{equation*}
for some smooth function $h_i$. In particular, $\frac{g_i(r, \theta)}{r}$ causes no trouble.
Best Answer
Multiply with $2\dot x$ and semi-integrate to find $$ \frac{d}{dt}(\dot x^2+x^2)=2\dot x^2(1-(\dot x^2+x^2)) $$ so it can seem as a good idea to set $r^2=\dot x^2+x^2$ and use the circle equation to define an angle $\theta$ so that $$ \dot x=r\cosθ,~~x=r\sinθ $$ where both $r$ and $θ$ are time-dependent functions.