Recall that the center manifold of a dynamical system at an equilibrium point is made of the orbits tangent to the center subspace at this point, where the center subspace is spanned by the eigenvectors of the linearized system at this point, corresponding to eigenvalues with real part zero.
Here one considers the equilibrium point $(0,0)$, the linearized system at this point is $$\dot x=0,\qquad \dot y=-y,$$ hence the two eigenvalues at $(0,0)$ are $0$ with eigenvector $(1,0)$, which yields the center subspace $\{(x,y)\mid y=0\}$, and $-1$ with eigenvector $(0,1)$, which yields the stable subspace $\{(x,y)\mid x=0\}$.
Regarding the center manifold $C$ at $(0,0)$, after some tedious computations and comparing coefficients of $x$ of both sides... one gets a rather different result, namely, in your notations, $$g(x)=x^2.$$ Thus, $C=\{(x,y)\mid y=x^2\}$ and the dynamics on $C$ is $$\dot x=-xg(x)=-x^3.$$ To solve question (b), change coordinates by considering $$y=g(x)\cdot z=x^2\cdot z,$$ which only excludes, as is natural, the stable manifold $S=\{(x,y)\mid x=0\}$. Then the $(x,z)$ differential system is $$\dot x=-x^3\cdot z,\qquad \dot z=-z+1,$$ in particular, $$z(t)\to1.$$ Thus, every solution of the system is attracted to the center manifold $C=\{(x,y)\mid y=g(x)\}$ in the sense that, for every initial condition $(x(0),y(0))$ not in the stable manifold $S=\{(x,y)\mid g(x)=0\}$, $$\lim_{t\to+\infty}\frac{y(t)}{g(x(t))}=1.$$
$\hspace{4cm}$
First note that the third equation has the solution
$$
z(t)^{-2}-z_0^{-2}=2t\implies z(t)=\frac{z_0}{\sqrt{1+2z_0^2t}}
$$
where now indeed $z(0)=z_0$.
Next, you can be a little creative in solving the second equation by inserting the third to reduce the 4th degree term,
$$
\dot y+2y=2z^2-2z^4=2z^2+2z\dot z=\frac{d}{dt}(z^2)+2z^2.
$$
This means that "luckily" the terms turn out to be that the same differential operator $(D+2)$ is applied to both sides. The solution then is
$$
y(t)-z(t)^2=(y_0-z_0^2)e^{-2t},\\
y(t)=\frac{z_0^2}{1+2z_0^2t}+(y_0-z_0^2)e^{-2t}.
$$
I do not see how there can be any unstable manifold. From points outside the set with $x_0=0$ and $y_0=z_0^2$ the solutions will converge exponentially fast towards points within that set, and then slowly along that set towards the origin. In my mind this was the characteristic of a center manifold?
Best Answer
The equilibrium is attained at the solutions for
$$ \cases{ y=0\\ -y+a x^2+b x y = 0 } $$
so $(0,0)$ is the equilibrium point. To qualify it we compute the jacobian at this point giving
$$ J = \left( \begin{array}{cc} 0 & 1 \\ 0 & -1 \\ \end{array} \right) $$
with eigenvalues $(1,\ 0)$ so the equilibrium manifold is one-dimensional. To find this manifold we proceed as follows.
For the dynamical system
$$ \cases{ \dot x=f(x,y)\\ \dot y=g(x,y) } $$
Proposing the solution
$$ y=h(x) = \sum_{k=1}^n a_k x^k $$
we have
$$ \dot y=h_x(x)\dot x = h_x(x)f(x,h(x))=g(x,h(x)) $$
assuming $n=4$ equating the $x$ powers we arrive at
$$ \left\{ \begin{array}{rcl} a_1&=&0 \\ a_2 &=& a \\ a_3 &=& a b-2 a^2\\ \end{array} \right. $$
and solving we have
$$ h(x) = a x^2+a(b-2a) x^3+ O(x^4) $$
as a near origin approximation.
Follows a plot showing the stream plot for $a = -\frac 12, b = 1$ showing in thick blue a near origin center manifold segment and in red dashed, a path beginning at $(0.5,0.5)$
NOTE
The central manifold approximate flow for $n=4$ is given by
$$ \dot x = h(x) = a x^2+a (b-2a) x^3+ O(x^4) $$