$xyxy = yxy^2 = yx^4 = y^3 x$ by post-multiplying $xyx$ by $y$.
$xyxy = x^2 yx$ by pre-multiplying $yxy$ by $x$.
Therefore $y^3 x = x^2 y x$, so (by cancelling $x$ from RHS) $y^3 = x^2 y$, so (by cancelling $y$ from RHS) $y^2 = x^2$.
Therefore $\underbrace{y^2 = x^3}_{\text{relation}} = x^2$, so $x = e$ and hence $y=e$.
You can rewrite your relator such that it has $0$ exponent sum in two of the generators, as the map $x\mapsto xyz, y\mapsto y, z\mapsto z$ is a Nielsen transformation:
$$
\begin{align*}
\langle x, y, z\mid x^{2}=y^2z^2\rangle
&\cong\langle x, y, z\mid x^{2}z^{-2}y^{-2}\rangle\\
&\cong\langle x, y, z\mid (xyz)^{2}z^{-2}y^{-2}\rangle
\end{align*}
$$
Under the abelinisation map we then get the group:
$$
\begin{align*}
\langle x, y, z\mid (xyz)^{2}z^{-2}y^{-2}\rangle^{ab}&=\langle x, y, z\mid x^2\rangle^{ab}\\
&\cong \mathbb{Z}^2\times(\mathbb{Z}/2\mathbb{Z})
\end{align*}
$$
This is a specific case of a more general phenomenon, where one can adapt the Euclidean algorithm to rewrite using automorphisms a word $W\in F(a, b, \ldots)$ such that it has zero exponent sum in all but one of the relators. For example, writing $\sigma_x$ for the exponent sum of the relator word in the letter $x$:
$$
\begin{align*}
&\langle a, b\mid a^6b^8\rangle&&\sigma_a=6, \sigma_b=8\\
&\cong\langle a, b\mid (ab^{-1})^6b^8\rangle&&\text{by applying}~a\mapsto ab^{-1}, b\mapsto b\\
&=\langle a, b\mid (ab^{-1})^5ab^7\rangle&&\sigma_a=6, \sigma_b=2\\
&\cong\langle a, b\mid (a(ba^{-3})^{-1})^5a(ba^{-3})^7\rangle&&\text{by applying}~a\mapsto a, b\mapsto ba^{-3}\\
&\cong\langle a, b\mid (a^4b^{-1})^5a(ba^{-3})^7\rangle&&\sigma_a=0, \sigma_b=2
\end{align*}
$$
You can think of this as a "non-commutative Smith normal form", but it is more useful in this context than the Smith normal form as it gives you more information than just the abelianisation. For example, it is used in the HNN-extension version of the Magnus hierarchy ($a$ is the stable letter, and the associated subgroups are free by the Freiheitssatz; see J. McCool and P. Schupp, On one relator groups and HNN extensions, Journal of the Australian Mathematical Society, Volume 16, Issue 2, September 1973 , pp. 249-256 doi).
Best Answer
Claim: $G\cong C_7\times C_3$. Start with the map from the free group on two generators $F_2$ to $C_7\times C_3$ given by $x\mapsto(1,0)$ and $y\mapsto (0,1)$. Verify that $x^7$, $y^3$, and $yxy^{-1}x^{-1}$ are in the kernel of this map, so we have an induced quotient $f\colon G\to C_7\times C_3$. By Von Dyck's theorem (or directly) we know $f$ is surjective. Now argue that $|G|$ is at most $21$: every element can be written $x^my^n$ with $0\le m\le 6$ and $0\le n\le 2$.