For any subset $|Y| \subseteq |X|$, there exists a monomorphism $\iota \colon Y \hookrightarrow X$ supported on $Y$. However, these do not have the property that any map landing in $|Y|$ factors through $Y$.
In fact, I will show (using results from the Samuel seminar on epimorphisms of rings [Sam], [Laz]) that in the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$ that you give, there is no minimal monomorphism it factors through.
Lemma 1. For any subset $|Y| \subseteq |X|$, there exists a monomorphism $\iota \colon Y \hookrightarrow X$ supported on $Y$.
Proof. Indeed, let $Y = \coprod_{y \in |Y|} \operatorname{Spec} \kappa(y)$. Then the natural map $\iota \colon Y \to X$ is a monomorphism, because $Y \times_X Y$ is just $Y$. (See [ML, Exc. III.4.4] for this criterion for monomorphism.) $\square$
Now we focus on the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$. Write $f \colon Z \to X$ for this map, and note that $f$ is an isomorphism over $D(x) \subseteq X$. Assume $\iota \colon Y \to X$ is a minimal immersion such that there exists a factorisation of $f$ as
$$Z \stackrel g\to Y \stackrel \iota \to X.$$
For any irreducible scheme $S$, write $\eta_S$ for its generic point. We denote the origin of $\mathbb A^2$ by $0$.
Lemma 2. We must have $|Y| = |\!\operatorname{im}(f)|$ or $|Y| = |\!\operatorname{im}(f)| \cup \{\eta_{V(x)}\}$, and $Y$ is integral.
Proof. Clearly $|Y|$ contains $|\!\operatorname{im}(f)|$. If this inclusion were strict, then $|Y|$ contains some point $y$ that is not in the image of $f$. If $y$ is closed in $X$, then the open immersion $Y \setminus \{y\} \to Y$ gives a strictly smaller monomorphism that $f$ factors through, contradicting the choice of $Y$. Thus, $y$ has to be the generic point of $V(x)$. This proves the first statement.
For the second, note that the scheme-theoretic image $\operatorname{im}(g)$ of $g$ is $Y$. Indeed, if it weren't, then replacing $Y$ by $\operatorname{im}(g)$ would give a smaller monomorphism factoring $f$, contradicting minimality of $Y$. But the scheme-theoretic image of an integral scheme is integral, proving the second statement. $\square$
We now apply the following two results from the Samuel seminar on epimorphisms of rings [Sam]:
Theorem. Let $\iota \colon Y \to X$ be a quasi-compact birational monomorphism of integral schemes, with $X$ normal and locally Noetherian. Then $\iota$ is flat.
Proof. See [Sam, Lec. 7, Cor. 3.6]. $\square$
If $U \subseteq Y$ is an affine open neighbourhood of $0$ and $R = \Gamma(U,\mathcal O_U)$, then we get a flat epimorphism $\phi \colon k[x,y] \to R$ of $k$-algebras (not necessarily of finite type).
Theorem. Let $f \colon A \to B$ be a flat epimorphism of rings, and assume $A$ is normal and $\operatorname{Cl}(A)$ is torsion. Then $f$ is a localisation, i.e. $B = S^{-1}A$ for $S \subseteq A$ a multiplicative subset.
Proof. See [Laz, Prop. IV.4.5]. $\square$
Thus, $R = S^{-1}k[x,y]$ for some multiplicative set $S \subseteq k[x,y]$. This implies that $V = X \setminus U$ is a union of (possibly infinitely many) divisors. Moreover, $V \cap D(x) \subseteq D(x)$ has finitely many components since $D(x)$ is Noetherian (and $\iota$ is an isomorphism over $D(x)$). But then $V \cap V(x)$ is either finite or all of $V(x)$. This contradicts the fact that $U \cap V(x)$ equals $\{0\}$ or $\{0,\eta_{V(x)}\}$ (depending whether $\eta_{V(x)} \in Y$). $\square$
References.
[Laz] D. Lazard, Autour de la platitude, Bull. Soc. Math. Fr. 97, 81-128 (1969). ZBL0174.33301.
[ML] S. Mac Lane, Categories for the working mathematician. Graduate Texts in Mathematics 5. New York-Heidelberg-Berlin: Springer-Verlag (1971). ZBL0232.18001.
[Sam] P. Samuel et al., Séminaire d’algèbre commutative (1967/68): Les épimorphismes d’anneaux. Paris: École Normale Supérieure de Jeunes Filles (1968). ZBL0159.00101.
Best Answer
It seems indeed that example 2.10 in the Stacks project morphisms of schemes chapter provides a counter example, where the sheaf of ideals of regular functions whose restrictions to U vanish on Z is not quasi-coherent, because if it were part (3) of lemma 4.3 would be fulfilled. Thank you again for this hint, and if I am not mistaken, I have answered my own question, and an errata should be added for this book on page 26, where the open immersion of U in X should be supposed quasi-compact.