As has been shown to OP already, this link gives a certain method to determine $G$.
If $\mathbf{G}=(G_1,G_2,G_3)$ can be decomposed into another potential $\mathbf{H}=(H_1,H_2,H_3)$ and the gradient of a scalar function $f(x,y,z)$, i.e
$$\mathbf{G} = \mathbf{H} + \nabla f$$
This implies that
$$ \nabla \times \mathbf{G} = \nabla \times (\mathbf{H} + \nabla f) = \nabla \times \mathbf{H} + \nabla \times(\nabla f) = \nabla \times \mathbf{H} + \mathbf{0} = \nabla \times \mathbf{H} $$
Hence $\mathbf{G}$ is not unique and one can make specific choices to determine $\mathbf{G}$.
If we make the choice such that $$ \frac{\partial f}{\partial z} = -H_3$$
Then $\mathbf{G}=(H_1,H_2,H_3) +(\frac{\partial f}{\partial x}, \frac{\partial f}{\partial y},-H_3) = (H_1+\frac{\partial f}{\partial x},H_2+\frac{\partial f}{\partial y},0) = (G_1,G_2,0)$.
So we can choose $\mathbf{G}$ such that it can be either
\begin{align}
&(0,G_2,G_3) \text{ or}\\
&(G_1,0,G_3) \text{ or}\\
&(G_1,G_2,0)
\end{align}
So let us see what these choices can produce. We have the equations of $\nabla \times \mathbf{G} = \mathbf{F}$,
\begin{align}
&\frac{\partial G_3}{\partial y} - \frac{\partial G_2}{\partial z} = -y \\
&\frac{\partial G_1}{\partial z} - \frac{\partial G_3}{\partial x} = x \\
&\frac{\partial G_2}{\partial x} - \frac{\partial G_1}{\partial y} = 0 \\
\end{align}
If $G_1 = 0$, then we have
\begin{align}
&\frac{\partial G_3}{\partial y} - \frac{\partial G_2}{\partial z} = -y \\
-&\frac{\partial G_3}{\partial x} = x \Rightarrow G_3 = -\frac{x^2}{2}+C_3(y,z)\\
&\frac{\partial G_2}{\partial x} = 0 \Rightarrow G_2 = C_2(y,z) \\
\end{align}
substituting the last two equations into the first, we get
$$\frac{\partial G_3}{\partial y} - \frac{\partial G_2}{\partial z} = [C_3(y,z)]_y - [C_2(y,z)]_z = -y$$
Here, for simplicity, we can choose $C_3(y,z)=0$, because if $C_2(y,z)=0$ then two components of $\mathbf{G}$ are $0$, which never happens in the given possiblities. So,
$$-[C_2(y,z)]_z = -y \Rightarrow C_2(y,z)=yz$$
Then $\boxed{\mathbf{G} = (0,-yz,-\frac{x^2}{2})}$ which can be verified to satisfy $\nabla \times \mathbf{G} = \mathbf{F}$
If $G_2 = 0$, then we have
\begin{align}
&\frac{\partial G_3}{\partial y} = -y \Rightarrow G_3 = -\frac{y^2}{2} + C_3(x,z)\\
&\frac{\partial G_1}{\partial z} - \frac{\partial G_3}{\partial x} = x \\
-&\frac{\partial G_1}{\partial y} = 0 \Rightarrow G_1 = C_1(x,z) \\
\end{align}
then we get
$$\frac{\partial G_1}{\partial z} - \frac{\partial G_3}{\partial x} = [C_1(x,z)]_z - [C_3(x,z)]_x = x$$
Here, for simplicity, we can choose $C_3(x,z)=0$, for the same reason being we dont want two components of $\mathbf{G}$ to be $0$
$$[C_1(x,z)]_z = x \Rightarrow C_1(x,z)=xz$$
Then $\boxed{\mathbf{G} = (xz,0,-\frac{y^2}{2})}$ which can be verified to satisfy $\nabla \times \mathbf{G} = \mathbf{F}$
If $G_3 = 0$, then we have
\begin{align}
-& \frac{\partial G_2}{\partial z} = -y \Rightarrow G_2 = yz + C_2(x,y)\\
&\frac{\partial G_1}{\partial z} = x \Rightarrow G_1 = xz +C_1(x,y) \\
&\frac{\partial G_2}{\partial x} - \frac{\partial G_1}{\partial y} = 0 \\
\end{align}
then we get
$$\frac{\partial G_2}{\partial x} - \frac{\partial G_1}{\partial y} = [C_2(x,y)]_x - [C_1(x,y)]_y = 0 \Rightarrow [C_2(x,y)]_x = [C_1(x,y)]_y $$
Here, for simplicity, we can choose $C_2(x,y)=C_1(x,y)=0$, and so
$\boxed{\mathbf{G} = (xz,yz,0)}$ which can be verified to satisfy $\nabla \times \mathbf{G} = \mathbf{F}$
Of the three boxed solutions, only $\mathbf{G} = (xz,0,-\frac{y^2}{2})$ is on our list, and hence it is our answer.
However, given that we have a list of options, one can arrive to this answer earlier by realizing that certain solutions of the form $(xz, f(y),0)$ will never satisfy the curl equation, so the answer will be in the form$(G_1,0,G_3)$
Best Answer
You don't have to find the integration constant immediately. Keep proceeding as follows.
After you determined that $f(x,y,z) = xy+g(y,z)$, differentiate with respect to $y$.
This gives $\frac{\partial f}{\partial y}=x+\frac{\partial g}{\partial y}=F_y=x$.
Thus, $\frac{\partial g}{\partial y}=0$, which implies that $g$ is a function of $z$ only. In turn, this means that $f(x,y,z)=xy+h(z)$.
Next, differentiate $f$ with respect to $z$.
This gives $\frac{\partial f}{\partial z}=h'(z)=F_z=z^2$.
Thus, $h(z)=\frac13z^3+C$.
Finally, $f(x,y,z)=xy+h(z)=xy+\frac13z^3+C$.
To check this, we have $$\vec F=\nabla f(x,y,z)$$
$$=\hat x\frac{\partial f}{\partial x}+\hat y\frac{\partial f}{\partial y}+\hat z\frac{\partial f}{\partial z}$$
$$=\hat xy+\hat yx+\hat zz^2$$which completes the task!