One way we could go is to subtract $x$ times the second equation from the first, getting us: $$xy^2 - 2y + 3x^2-x(x^2y + 2x + y^2) = 0\\xy^2 - 2y + 3x^2-x^3y - 2x^2 -xy^2 = 0\\x^2-x^3y-2y=0\\x^2=x^3y+2y\\x^2=(x^3+2)y$$ Note that if we had $x^3+2=0,$ then the last equation above would become $x^2=0,$ so $x=0,$ contradicting our assumption that $x\ne 0.$ Thus, we don't have to worry about $x^3+2$ being zero, and so $$y=\frac{x^2}{x^3+2}.\tag{$\star$}$$ Substituting for $y$ in the second equation gets us $$x^2\cdot\frac{x^2}{x^3+2}+2x+\left(\frac{x^2}{x^3+2}\right)^2=0,$$ or $$x\left(\frac{x^3}{x^3+2}+\frac{x^3}{(x^3+2)^2}+2\right)=0,$$ which (since $x\ne 0$) is equivalent to $$\frac{x^3}{x^3+2}+\frac{x^3}{(x^3+2)^2}+2=0\\x^3(x^3+2)+x^3+2(x^3+2)^2=0\\x^6+3x^3+2(x^6+4x^3+4)=0\\3x^6+11x^3+8=0\\3\left(x^3\right)^2+11x^3+8=0\\x^3=\frac{-11\pm\sqrt{121-96}}6\\x^3=\frac{-11\pm \sqrt{25}}6\\x^3=\frac{-11\pm 5}6\\x^3=-1\textrm{ or }x^3=-\frac83$$
If we're looking for real solutions, then we have $x=-1$ or $x=-\frac{2}{\sqrt[3]3},$ but if we're interested in non-real solutions, too, then since $-\frac12\pm\frac{\sqrt3}2$ are cube roots of $-1,$ too, we also have $x=\frac12\pm\frac{\sqrt3}2$ and $x=-\frac{2}{\sqrt[3]3}\left(-\frac12\pm\frac{\sqrt3}2\right).$ (For brevity, I will denote $\omega=-\frac12+\frac{\sqrt3}2$ and $\overline\omega=-\frac12-\frac{\sqrt3}2.$)
When $x\in\left\{-1,\omega,\overline\omega\right\},$ we have $x^3=-1,$ so $x^3+2=1,$ and so $(\star)$ becomes $$y=x^2.$$ Thus, we obtain the solutions $(-1,1),$ $\left(\omega,\omega^2\right)=\left(\omega,-\overline\omega\right),$ and $\left(\overline\omega,\overline\omega^2\right)=\left(\overline\omega,-\omega\right).$
When $x\in\left\{-\frac{2}{\sqrt[3]3},-\frac{2\omega}{\sqrt[3]3},-\frac{2\overline\omega}{\sqrt[3]3}\right\},$ we have $x^3=-\frac83,$ so $x^3+2=-\frac23,$ and so $(\star)$ becomes $$y=-\frac32x^2.$$ Thus, we obtain the solutions $\left(-\frac{2}{\sqrt[3]3},-2\sqrt[3]3\right),$ $\left(-\frac{2\omega}{\sqrt[3]3},2\overline\omega\sqrt[3]3\right),$ and $\left(-\frac{2\overline\omega}{\sqrt[3]3},2\omega\sqrt[3]3\right).$
In summary, our solutions are: $$(0,0),(-1,1),\left(-\frac{2}{\sqrt[3]3},-2\sqrt[3]3\right),\left(\omega,-\overline\omega\right),\left(\overline\omega,-\omega\right),\left(-\frac{2\omega}{\sqrt[3]3},2\overline\omega\sqrt[3]3\right),\left(-\frac{2\overline\omega}{\sqrt[3]3},2\omega\sqrt[3]3\right),$$ with the first three being the real solutions.
$(x \le 3y \ne -x)$
Let $a = \dfrac{2}{x^2} > 0$ and $b = \dfrac{3y}{x}$. The system of equation becomes
$$\left\{ \begin{align} \left(\frac{2}{a} - 1\right)^2 + 3 = \frac{2b \cdot \left(\dfrac{2}{a}\right)^3}{\dfrac{2}{a} + 2}\\ b - 1 = \sqrt{\frac{2a - b - b^2}{1 + b}} \end{align} \right.$$
$$\iff \left\{ \begin{align} (2 - a)^2 + 3a^2 = \frac{8b}{a + 1}\\ (b - 1)^2 + b = \frac{2a}{b + 1} \end{align} \right.$$
$$\iff \left\{ \begin{align} a^2 - a + 1 = \frac{2b}{a + 1}\\ b^2 - b + 1 = \frac{2a}{b + 1} \end{align} \right.$$
$$\iff \left\{ \begin{align} a^3 + 1 = 2b\\ b^3 + 1 = 2a \end{align} \right.$$
$\iff a^3 - 2b = b^3 - 2a \iff (a - b)(a^2 + ab + b^2 + 2) = 0$
However $a^2 + ab + b^2 + 2 \ge 2 > 0, \forall a, b \in \mathbb R$
$\implies a^3 - 2a + 1 = b^3 - 2b + 1 = 0$ and $a = b$
$\iff (a - 1)(a^2 + a - 1) = (b - 1)(b^2 + b - 1) = 0$ and $a = b$
$\iff a = b = 1$ or $a = b = \dfrac{\sqrt 5 - 1}{2} (a > 0)$
$\implies \dfrac{2}{x^2} = \dfrac{3x}{y} = 1$ or $\dfrac{2}{x^2} = \dfrac{3x}{y} = \dfrac{\sqrt 5 - 1}{2}$
$\implies (x, y) = \left(\pm \sqrt 2, \pm \dfrac{\sqrt 2}{3}\right)$ or $(x, y) = \left(\pm \sqrt{\sqrt 5 + 1}, \pm \dfrac{\sqrt{\sqrt 5 + 1}\left(\sqrt 5 - 1\right)}{6}\right)$
Nevertheless $x \le 3y$, $\implies \left(- \sqrt{\sqrt 5 + 1}, - \dfrac{\sqrt{\sqrt 5 - 1}}{3}\right)$ is a solution but $\left(\sqrt{\sqrt 5 + 1}, \dfrac{\sqrt{\sqrt 5 - 1}}{3}\right)$ is not.
Best Answer
Let $x' = x + 1$ and $y' = y - 1$, the system of equations becomes
$$\left\{ \begin{align} \sqrt{y'^2 - 8x' + 2y' + 18} - \sqrt[3]{x'y' - 5x' - y' + 17} = 1\\ \sqrt{2(x' - y')^2 + 2x' + 2y' + 4} - \sqrt{y' + 1} = \sqrt{x' + 1}\end{align} \right.$$
Solving the second equation, we have that $$\sqrt{2(x' - y')^2 + 2x' + 2y' + 4} \ge \sqrt{2[(x' + 1) + (y' + 1)]} \ge \sqrt{x' + 1} + \sqrt{y' + 1}$$
The equality sign occurs when $x' = y'$. Substituting $x' = y'$ into the first equation, we have that
$$\sqrt{x'^2 - 6x' + 18} - \sqrt[3]{x'^2 - 6x' + 17} = 1$$
Let $\sqrt[3]{x'^2 - 6x' + 17} = t (t \ge 2)$, we have that $$\sqrt{t^3 + 1} - t = 1 \iff \sqrt{t^2 - t + 1} = \sqrt{t + 1} \iff t^2 - 2t = 0 \implies t = 2$$
$$x'^2 - 6x' + 9 = 0\implies x' = y' = 3 \implies (x, y) = (2, 4)$$.