This answer posted in response to the modified system, given in (1) and (2) below:
Oy Gevalt! NOW there is work!
Where is Moshe now that we need him to lead us out from under Pharoah's toil?
Well, I ain't no $Moshe$ but I can cut some down some of the work, thus:
The system to be considered is now
$x' = x^2 + y^2 - 1, \tag{1}$
$y' = x^2 - y^2; \tag{2}$
the number of equilibria has jumped from none to four! To see this, note that now $x' = y' = 0$ implies, from (1), (2), that
$x^2 + y^2 = 1, \tag{3}$
$x^2 = y^2; \tag{4}$
using (4) in (3) yields
$2x^2 = 1 \Rightarrow x = \pm \dfrac{\sqrt 2}{2}; \tag{5}$
we see from (4) that $y$ may take the same values; the equilibria occur at the four points
$(x, y) = (\pm \dfrac{\sqrt 2}{2}, \pm \dfrac{\sqrt 2}{2}). \tag{6}$
It may also be seen, geoemtrically, that the equilibria are given by (6), since (3) is the equation of a circle, centered at the origin and of radius $1$, and (4) is the combined equation of the two lines $x \pm y = 0$, since
$x^2 = y^2 \Rightarrow x = \pm y \Rightarrow x \pm y = 0; \tag{7}$
the circle intersects these lines at the specified points (6). In any event, having the equilibria of the system (1)-(2) at hand, the next step is to linearize the equations about these four points, and see what we get. Linearizing requires computation of the Jacobian matrix $J(x, y)$ of the vector field $(x', y')^T = (x^2 + y^2 -1, x^2 - y^2)^T$; we have
$J(x, y) = \begin{bmatrix} \dfrac{\partial x'}{\partial x} & \dfrac{\partial x'}{\partial y} \\ \dfrac{\partial y'}{\partial x} & \dfrac{\partial y'}{\partial y} \end{bmatrix} = \begin{bmatrix} 2x & 2y \\ 2x & -2y \end{bmatrix} = 2\begin{bmatrix} x & y \\ x & -y \end{bmatrix}, \tag{8}$
and we next must evaluate and eigen-analyze $J(\pm \dfrac{\sqrt{2}}{2}, \pm \dfrac{\sqrt{2}}{2})$ for all four possible combinations of $\pm \dfrac{\sqrt{2}}{2}$, i.e., at all four points $(\pm \dfrac{\sqrt{2}}{2}, \pm \dfrac{\sqrt{2}}{2})$. Well, some the the Oy! Gevalt! can be mollified by lessening the amount of work by realizing that, due to certain symmetries of the problem, there are really only two matrices $J(x, y)$ which need to be considered, not four. This is most easily seen by breaking the situation up into quadrants:
$J(-\dfrac{\sqrt{2}}{2}, -\dfrac{\sqrt{2}}{2}) = -J(\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2}), \; \; \text{for quadrants I, III}, \tag{9}$
$J(-\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2}) = -J(\dfrac{\sqrt{2}}{2}, -\dfrac{\sqrt{2}}{2}), \; \; \text{for quadrants II, IV}; \tag{10}$
in this way we only need perform the eigen-analysis on two matrices out of the four $J(\pm \dfrac{\sqrt{2}}{2}, \pm \dfrac{\sqrt{2}}{2})$, so lets start with $J_I = J(\dfrac {\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2})$; its characteristic polynomial, call it $p_I(\lambda)$, is
$p_I(\lambda) = \det (\begin{bmatrix} \sqrt{2} - \lambda & \sqrt{2} \\ \sqrt{2} & -\sqrt{2} - \lambda \end{bmatrix}) = \lambda^2 - 4. \tag{11}$
We see from (11) that the eigenvalues of $J_I$ are $\lambda = \pm 2$; since $J_I$ has a positive eigenvalue, the point $(\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2})$ is unstable; since $J_I$ also has a negative eigenvalue, this point is a saddle; by (9), $J_{III} = -J_I$ also has eigenvalues $\pm 2$, and hence also unstable and a saddle. These facts are borne out by the excellent graphic contributed by Amzoti in his answer. We next turn to $J_{II}$; since we are now in the second quadrant, we have
$J_{II} = J(-\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2}) = \begin{bmatrix} -\sqrt{2} & \sqrt{2} \\ -\sqrt{2} & -\sqrt{2}\end{bmatrix}, \tag{12}$
$p_{II}(\lambda) = \det(J_{II} - \lambda I) = \det(\begin{bmatrix} -\sqrt{2} - \lambda & \sqrt{2} \\ -\sqrt{2} & -\sqrt{2} - \lambda \end{bmatrix})$
$= (\lambda + \sqrt{2})^2 + 2 = \lambda^2 + 2\sqrt{2} \lambda + 4; \tag{13}$
the zeroes of $p_{II}(\lambda)$ are found via the quadratic formula:
$\lambda = \dfrac{1}{2}(-2\sqrt{2} \pm \sqrt{8 - 16}) = \dfrac{1}{2}(-2\sqrt{2} \pm 2i\sqrt{2}) = -\sqrt{2} \pm i\sqrt{2}; \tag{14}$
we we see that the eigenvalues of $J_{II}$ are not real, but have negative real part; thus $(-\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2})$ is a stable spiral. Furthermore, since $J_{IV} = -J_{II}$, the eigevalues of $J_{IV}$ are $\lambda = \sqrt{2} \pm i\sqrt{2}$; thus the point $(-\dfrac{\sqrt{2}}{2}, -\dfrac{\sqrt{2}}{2})$ is an unstable spiral. All these computations support and are supported by Amzoti's grahic of the phase portrait of (1)-(2).
The stable spiral point at $(-\dfrac{\sqrt{2}}{2}, \dfrac{\sqrt{2}}{2})$ is in fact asymptotically stable; this follows from the fact that $\Re(\lambda) < 0$ for each of the eigenvalues of $J_{II}$; this fact is both well-known and well documented, for example in the excellent reference provided by Amzoti in his comment.
There appears to be a discrepancy in the calculation of eigenvalues by Monolinte and myself, but we agree on the qualitative features of the equilibrium points. As ever, abstract analysis is easy but arithmetic proves difficult! Until further notice, I'm standing by my calculations.
Hope this helps! Cheers,
and as ever,
Fiat Lux!!!
Best Answer
This is a powerful qualitative tool that helps us get a general understanding of the behavior. When the DEQ is not solvable (other than using numerical methods or otherwise), this is the next best thing - we do qualitative analyses. Just look at how much information you were able to garner just from the direction field and plotting typical solutions using numerical methods on top of those.
We have to just look at the derivatives sign or if it equals zero to determine if it is a 'sink', 'source' or 'node' and can draw a phase line from this, that correlates to the nice phase plot you gave.
Theorem (Stability and Instability Conditions)
Let $f$ and $f′$ be continuous.
The equation $y′ = f(y)$ has a sink at $y = y_0$ provided $f(y_0) = 0$ and $f′(y_0) \lt 0$ (stable).
An equilibrium $y = y_1$ is a source provided $f(y_1) = 0$ and $f′(y_1) \gt 0$ (unstable).
There is no test when $f′$ is zero at an equilibrium (indeterminate). The no-test case can sometimes be decided by an additional test:
(a) Equation $y′ = f(y)$ has a sink at $y = y_0$ provided $f(y)$ changes sign from positive to negative at $y = y_0$.
In your example, $y' = f(y) = y - \sqrt{y}$, which is an autonomous system.
You already know the equilibrium points, now, use the theorem and then compare this qualitative method with what you already have in the diagram.
Regards