One way we could go is to subtract $x$ times the second equation from the first, getting us: $$xy^2 - 2y + 3x^2-x(x^2y + 2x + y^2) = 0\\xy^2 - 2y + 3x^2-x^3y - 2x^2 -xy^2 = 0\\x^2-x^3y-2y=0\\x^2=x^3y+2y\\x^2=(x^3+2)y$$ Note that if we had $x^3+2=0,$ then the last equation above would become $x^2=0,$ so $x=0,$ contradicting our assumption that $x\ne 0.$ Thus, we don't have to worry about $x^3+2$ being zero, and so $$y=\frac{x^2}{x^3+2}.\tag{$\star$}$$ Substituting for $y$ in the second equation gets us $$x^2\cdot\frac{x^2}{x^3+2}+2x+\left(\frac{x^2}{x^3+2}\right)^2=0,$$ or $$x\left(\frac{x^3}{x^3+2}+\frac{x^3}{(x^3+2)^2}+2\right)=0,$$ which (since $x\ne 0$) is equivalent to $$\frac{x^3}{x^3+2}+\frac{x^3}{(x^3+2)^2}+2=0\\x^3(x^3+2)+x^3+2(x^3+2)^2=0\\x^6+3x^3+2(x^6+4x^3+4)=0\\3x^6+11x^3+8=0\\3\left(x^3\right)^2+11x^3+8=0\\x^3=\frac{-11\pm\sqrt{121-96}}6\\x^3=\frac{-11\pm \sqrt{25}}6\\x^3=\frac{-11\pm 5}6\\x^3=-1\textrm{ or }x^3=-\frac83$$
If we're looking for real solutions, then we have $x=-1$ or $x=-\frac{2}{\sqrt[3]3},$ but if we're interested in non-real solutions, too, then since $-\frac12\pm\frac{\sqrt3}2$ are cube roots of $-1,$ too, we also have $x=\frac12\pm\frac{\sqrt3}2$ and $x=-\frac{2}{\sqrt[3]3}\left(-\frac12\pm\frac{\sqrt3}2\right).$ (For brevity, I will denote $\omega=-\frac12+\frac{\sqrt3}2$ and $\overline\omega=-\frac12-\frac{\sqrt3}2.$)
When $x\in\left\{-1,\omega,\overline\omega\right\},$ we have $x^3=-1,$ so $x^3+2=1,$ and so $(\star)$ becomes $$y=x^2.$$ Thus, we obtain the solutions $(-1,1),$ $\left(\omega,\omega^2\right)=\left(\omega,-\overline\omega\right),$ and $\left(\overline\omega,\overline\omega^2\right)=\left(\overline\omega,-\omega\right).$
When $x\in\left\{-\frac{2}{\sqrt[3]3},-\frac{2\omega}{\sqrt[3]3},-\frac{2\overline\omega}{\sqrt[3]3}\right\},$ we have $x^3=-\frac83,$ so $x^3+2=-\frac23,$ and so $(\star)$ becomes $$y=-\frac32x^2.$$ Thus, we obtain the solutions $\left(-\frac{2}{\sqrt[3]3},-2\sqrt[3]3\right),$ $\left(-\frac{2\omega}{\sqrt[3]3},2\overline\omega\sqrt[3]3\right),$ and $\left(-\frac{2\overline\omega}{\sqrt[3]3},2\omega\sqrt[3]3\right).$
In summary, our solutions are: $$(0,0),(-1,1),\left(-\frac{2}{\sqrt[3]3},-2\sqrt[3]3\right),\left(\omega,-\overline\omega\right),\left(\overline\omega,-\omega\right),\left(-\frac{2\omega}{\sqrt[3]3},2\overline\omega\sqrt[3]3\right),\left(-\frac{2\overline\omega}{\sqrt[3]3},2\omega\sqrt[3]3\right),$$ with the first three being the real solutions.
Best Answer
These are symmetric equations, so denoting $s=x+y,p=xy$ we have $$s^2-2p=2p^2$$ $$s(1+p)=4p^2$$ Now eliminate $s$: $$s^2=2p^2+2p=\left(\frac{4p^2}{1+p}\right)^2$$ $$2p(1+p)^3=16p^4$$ If $p=0$ then $s=0$ and obviously $x=y=0$. Otherwise, divide by $p$: $$(1+p)^3=(2p)^3$$ Assuming we are only solving in the reals: $$1+p=2p\qquad p=1,s=2$$ By Viète's formulas, $x$ and $y$ are the roots of $t^2-2t+1$, whereby we get $x=y=1$.