Not quite an answer, but too long for a comment. Combining the two (open/closed) gluing constructions, we can conclude that if $Z$ is an open subscheme of a closed subscheme $W$ of $X$, then it is an equalizer: this is Tom's first comment. Now my point is that not every locally closed subscheme is of this form.
Here is an example, which I see as a potential counterexample for Martin's question. Let $k$ be a field. Put $X=\mathrm{Spec}\,k[(T_n)_{n\in\mathbb{N}_{>0}}]$ and let $U$ be the complement of the origin $x$. For each $n>0$, let $Z_n\subset U$ be the $n$-th infinitesimal neighborhood of the "$n$-th basis vector" $e_n$. Put $Z=\bigcup_n Z_n$. This is a closed subscheme of $U$ (isomorphic as a scheme to $\coprod_n Z_n$) since if you invert $X_n$ you just get $Z_n$. But $Z$ is scheme-theoretically dense in $X$: if $f$ is a nonzero polynomial of degree $d$, its order of vanishing at $e_n$ cannot exceed $d$, hence the zero scheme of $f$ does not contain any $Z_{n}$ with $n>d$. So, the only closed subscheme of $X$ containing $Z$ is $X$, in which $Z$ is not open.
Remark: this implies that if $\mathcal{I}\subset \mathcal{O}_U$ is the ideal sheaf of $Z$, and $j:U\to X$ is the inclusion, then $j_\ast\mathcal{I}$ is not quasicoherent. In fact, the above argument applied to rational functions instead of polynomials shows that the stalk $(j_\ast\mathcal{I})_x$ is zero.
Presumably you meant to assume the schemes are finite type over $k$. To work naturally with orbit questions for such schemes one just has to bring in appropriate use of flatness to adapt intuition and experience from the traditional smooth setting over algebraically closed fields (e.g., one uses the robust theory of quotients modulo possibly non-smooth closed subgroup schemes).
For the orbit of a $k$-point $x \in X$ everything is fine: one has the orbit morphism $G \rightarrow X$, and its constructible image is locally closed because this can be checked over $\overline{k}$ (where we can pass to underlying reduced schemes). Likewise, for such $x$ one has the (possibly not smooth) stabilizer scheme $G_x$ inside $G$ over $k$ and a well-defined monomorphism $j_x:G/G_x \rightarrow X$. This is a locally closed immersion when $G$ is smooth (so in such cases we denote its smooth locally closed image with reduced structure as $Gx$, onto which the orbit map from $G$ is faithfully flat). Indeed, by descent theory we can extend the ground field to be algebraically closed, and then if $Z$ is the locally closed orbit with reduced structure we see that the orbit map factors through $Z$ and as such $G \rightarrow Z$ must be generically flat and hence faithfully flat, whence $j_x$ is an isomorphism onto $Z$ by descent theory.
Mere surjectivity of an orbit map (without flatness) is not a good notion of "homogeneous space".
In general with $x \in X(k)$ there is an absolutely huge gulf between $(Gx)(k)$ and $G(k)x$ (e.g., think about a surjective homomorphism between smooth affine groups!). In effect, say assuming $G$ is smooth so that the orbit is $G/G_x$ as shown above, you're asking about the existence of $k$-points in fibers of $\pi:G \rightarrow G/G_x$ over $k$-points of the target. Since $\pi$ is a $G_x$-torsor for the fppf topology, the obstructions lie in the fppf cohomology set ${\rm{H}}^1(k, G_x)$ (almost by definition), and in favorable cases one can analyze that by using knowledge of the structure of $G_x$.
Note however that considering the "orbit" of a point that is not a $k$-point doesn't quite make sense and is asking for trouble unless one is absolutely clear about definitions (and usually it is not what you want to ever consider -- it is technically usually much better to simply extend the ground field to work with orbits through rational points, and to use descent theory to go back down if you wish). However, there is a kind of substitute. You could contemplate $G$-stable non-empty locally closed subschemes $Z$ of $X$ such that the $G(\overline{k})$-action on $Z(\overline{k})$ is transitive and the surjective orbit map $G_{\overline{k}} \rightarrow Z_{\overline{k}}$ through some (equivalently, any) $z \in Z(\overline{k})$ is flat or equivalently identifies $Z_{\overline{k}}$ with $G_{\overline{k}}/(G_{\overline{k}})_z$. (If $G$ is smooth and $Z$ is geometrically reduced then this is equivalent to transitivity of the action on $\overline{k}$-points, for reasons explained above.) In effect, such $Z$ could be regarded as (non-empty) homogeneous spaces for $G$ inside $X$, and as an "orbit" for $G$ through any of closed point of $Z$ (it is easily checked that such a $Z$ is uniquely determined by any single closed point $x$ that it contains, but using the notation $Gx$ for such a $Z$ is going to lead to mistakes if $x \not\in X(k)$, so don't do that). If $Z(k)$ is empty then one should tread carefully to call such $Z$ an "orbit"; better to just say "homogeneous space".
Of course, for this viewpoint of homogeneous spaces inside $X$ (possibly without $k$-points) to be useful, say assuming $G$ is smooth to avoid some confusion, you want to prove an existence property: for any closed point $x \in X$ is there such a (necessarily smooth) $Z$ through $x$, or in other words does the orbit of $G_{\overline{k}}$ through a $\overline{k}$-point over $x$
descend to a locally closed subscheme of $X$?
The existence of such $Z$ fails most of the time when $G$ is not connected and $k$ is not algebraically closed. For example, let $G = \mu_n$ acting on $X = {\rm{GL}}_1$ by scaling with ${\rm{char}}(k) \nmid n$ and suppose $x \in X$ is a closed point such that $[k(x):k]$ does not divide $n$. In such cases, there is no way for $x$ to lie in a $k$-finite closed subscheme of $X$ of order dividing $n$, so there is no meaningful notion of $Gx$ over $k$. Even if $k$ is separably closed but not algebraically closed (say with characteristic $p > 0$) you're going to have problems when $G$ is finite etale since such a $Z$ would have to also be finite etale and hence cannot be found containing $x$ such that $k(x)$ is not separable over $k$ (so you get more counterexamples when $X = \mathbf{A}^1_k$ on which $G = \mathbf{Z}/(p)$ acts by translation, even when $[k(x):k] = p$ so one cannot use the more elementary degree obstruction).
In view of those basic counterexamples, it is an interesting exercise with Galois descent to prove that such a $Z$ does exist when $k(x)$ is separable over $k$ provided that the smooth $G$ is also connected. (The fun part is to figure out where you use connectedness.) If $k(x)$ is not separable over $k$ (and $G$ is smooth and connected) then this can fail. For instance, if $k$ is imperfect with characteristic $p$ and $t \in k - k^p$ then $G := \{v^p = u - tu^p\}$ is a smooth connected 1-dimensional $k$-subgroup of $\mathbf{G}_a^2$ and its Zariski closure $X$ in $\mathbf{P}^2_k$ has 1 point $x$ on the line at $\infty$, with $k(x) = k(t)$, so the $G$-action on $\mathbf{P}^2_k$ via $(u,v).[a,b,c] = [a + uc, b + vc, c]$ restricts to an action on $X$ in which $X - \{x\} = G$ with the translation action. There is no reasonable sense in which the $G$-stable complement $\{x\}$ of $X - \{x\}$ in $X$ can be regarded as a homogeneous space for $G$ since it is not $k$-smooth (and in particular, there is no $Z$ as above in this case).
Best Answer
For any subset $|Y| \subseteq |X|$, there exists a monomorphism $\iota \colon Y \hookrightarrow X$ supported on $Y$. However, these do not have the property that any map landing in $|Y|$ factors through $Y$.
In fact, I will show (using results from the Samuel seminar on epimorphisms of rings [Sam], [Laz]) that in the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$ that you give, there is no minimal monomorphism it factors through.
Proof. Indeed, let $Y = \coprod_{y \in |Y|} \operatorname{Spec} \kappa(y)$. Then the natural map $\iota \colon Y \to X$ is a monomorphism, because $Y \times_X Y$ is just $Y$. (See [ML, Exc. III.4.4] for this criterion for monomorphism.) $\square$
Now we focus on the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$. Write $f \colon Z \to X$ for this map, and note that $f$ is an isomorphism over $D(x) \subseteq X$. Assume $\iota \colon Y \to X$ is a minimal immersion such that there exists a factorisation of $f$ as $$Z \stackrel g\to Y \stackrel \iota \to X.$$ For any irreducible scheme $S$, write $\eta_S$ for its generic point. We denote the origin of $\mathbb A^2$ by $0$.
Proof. Clearly $|Y|$ contains $|\!\operatorname{im}(f)|$. If this inclusion were strict, then $|Y|$ contains some point $y$ that is not in the image of $f$. If $y$ is closed in $X$, then the open immersion $Y \setminus \{y\} \to Y$ gives a strictly smaller monomorphism that $f$ factors through, contradicting the choice of $Y$. Thus, $y$ has to be the generic point of $V(x)$. This proves the first statement.
For the second, note that the scheme-theoretic image $\operatorname{im}(g)$ of $g$ is $Y$. Indeed, if it weren't, then replacing $Y$ by $\operatorname{im}(g)$ would give a smaller monomorphism factoring $f$, contradicting minimality of $Y$. But the scheme-theoretic image of an integral scheme is integral, proving the second statement. $\square$
We now apply the following two results from the Samuel seminar on epimorphisms of rings [Sam]:
Proof. See [Sam, Lec. 7, Cor. 3.6]. $\square$
If $U \subseteq Y$ is an affine open neighbourhood of $0$ and $R = \Gamma(U,\mathcal O_U)$, then we get a flat epimorphism $\phi \colon k[x,y] \to R$ of $k$-algebras (not necessarily of finite type).
Proof. See [Laz, Prop. IV.4.5]. $\square$
Thus, $R = S^{-1}k[x,y]$ for some multiplicative set $S \subseteq k[x,y]$. This implies that $V = X \setminus U$ is a union of (possibly infinitely many) divisors. Moreover, $V \cap D(x) \subseteq D(x)$ has finitely many components since $D(x)$ is Noetherian (and $\iota$ is an isomorphism over $D(x)$). But then $V \cap V(x)$ is either finite or all of $V(x)$. This contradicts the fact that $U \cap V(x)$ equals $\{0\}$ or $\{0,\eta_{V(x)}\}$ (depending whether $\eta_{V(x)} \in Y$). $\square$
References.
[Laz] D. Lazard, Autour de la platitude, Bull. Soc. Math. Fr. 97, 81-128 (1969). ZBL0174.33301.
[ML] S. Mac Lane, Categories for the working mathematician. Graduate Texts in Mathematics 5. New York-Heidelberg-Berlin: Springer-Verlag (1971). ZBL0232.18001.
[Sam] P. Samuel et al., Séminaire d’algèbre commutative (1967/68): Les épimorphismes d’anneaux. Paris: École Normale Supérieure de Jeunes Filles (1968). ZBL0159.00101.