For any subset $|Y| \subseteq |X|$, there exists a monomorphism $\iota \colon Y \hookrightarrow X$ supported on $Y$. However, these do not have the property that any map landing in $|Y|$ factors through $Y$.
In fact, I will show (using results from the Samuel seminar on epimorphisms of rings [Sam], [Laz]) that in the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$ that you give, there is no minimal monomorphism it factors through.
Lemma 1. For any subset $|Y| \subseteq |X|$, there exists a monomorphism $\iota \colon Y \hookrightarrow X$ supported on $Y$.
Proof. Indeed, let $Y = \coprod_{y \in |Y|} \operatorname{Spec} \kappa(y)$. Then the natural map $\iota \colon Y \to X$ is a monomorphism, because $Y \times_X Y$ is just $Y$. (See [ML, Exc. III.4.4] for this criterion for monomorphism.) $\square$
Now we focus on the example $\mathbb A^2 \to \mathbb A^2$ given by $(x,y) \mapsto (x,xy)$. Write $f \colon Z \to X$ for this map, and note that $f$ is an isomorphism over $D(x) \subseteq X$. Assume $\iota \colon Y \to X$ is a minimal immersion such that there exists a factorisation of $f$ as
$$Z \stackrel g\to Y \stackrel \iota \to X.$$
For any irreducible scheme $S$, write $\eta_S$ for its generic point. We denote the origin of $\mathbb A^2$ by $0$.
Lemma 2. We must have $|Y| = |\!\operatorname{im}(f)|$ or $|Y| = |\!\operatorname{im}(f)| \cup \{\eta_{V(x)}\}$, and $Y$ is integral.
Proof. Clearly $|Y|$ contains $|\!\operatorname{im}(f)|$. If this inclusion were strict, then $|Y|$ contains some point $y$ that is not in the image of $f$. If $y$ is closed in $X$, then the open immersion $Y \setminus \{y\} \to Y$ gives a strictly smaller monomorphism that $f$ factors through, contradicting the choice of $Y$. Thus, $y$ has to be the generic point of $V(x)$. This proves the first statement.
For the second, note that the scheme-theoretic image $\operatorname{im}(g)$ of $g$ is $Y$. Indeed, if it weren't, then replacing $Y$ by $\operatorname{im}(g)$ would give a smaller monomorphism factoring $f$, contradicting minimality of $Y$. But the scheme-theoretic image of an integral scheme is integral, proving the second statement. $\square$
We now apply the following two results from the Samuel seminar on epimorphisms of rings [Sam]:
Theorem. Let $\iota \colon Y \to X$ be a quasi-compact birational monomorphism of integral schemes, with $X$ normal and locally Noetherian. Then $\iota$ is flat.
Proof. See [Sam, Lec. 7, Cor. 3.6]. $\square$
If $U \subseteq Y$ is an affine open neighbourhood of $0$ and $R = \Gamma(U,\mathcal O_U)$, then we get a flat epimorphism $\phi \colon k[x,y] \to R$ of $k$-algebras (not necessarily of finite type).
Theorem. Let $f \colon A \to B$ be a flat epimorphism of rings, and assume $A$ is normal and $\operatorname{Cl}(A)$ is torsion. Then $f$ is a localisation, i.e. $B = S^{-1}A$ for $S \subseteq A$ a multiplicative subset.
Proof. See [Laz, Prop. IV.4.5]. $\square$
Thus, $R = S^{-1}k[x,y]$ for some multiplicative set $S \subseteq k[x,y]$. This implies that $V = X \setminus U$ is a union of (possibly infinitely many) divisors. Moreover, $V \cap D(x) \subseteq D(x)$ has finitely many components since $D(x)$ is Noetherian (and $\iota$ is an isomorphism over $D(x)$). But then $V \cap V(x)$ is either finite or all of $V(x)$. This contradicts the fact that $U \cap V(x)$ equals $\{0\}$ or $\{0,\eta_{V(x)}\}$ (depending whether $\eta_{V(x)} \in Y$). $\square$
References.
[Laz] D. Lazard, Autour de la platitude, Bull. Soc. Math. Fr. 97, 81-128 (1969). ZBL0174.33301.
[ML] S. Mac Lane, Categories for the working mathematician. Graduate Texts in Mathematics 5. New York-Heidelberg-Berlin: Springer-Verlag (1971). ZBL0232.18001.
[Sam] P. Samuel et al., Séminaire d’algèbre commutative (1967/68): Les épimorphismes d’anneaux. Paris: École Normale Supérieure de Jeunes Filles (1968). ZBL0159.00101.
Best Answer
The abelian category of quasicoherent sheaves on a schemes determine the scheme. This is an old result of Gabriel ("des categories abeliennes" 1962), proved in full generality by Rosenberg. This means that, $\operatorname{QCoh}(X)$ does not only tell you the open subschemes of $X$ but also gives you the structure sheaf! I've known this result for some time but I had never looked at it in detail until today. I'll sketch what I have just learned hoping not to make big mistakes...
An abelian subcategory $B$ of an abelian category $A$ is said to be a thick subcategory if it is full and for any exact sequence in $A$
$$0\to M'\to M \to M''\to 0,$$
$M$ belongs to $B$ if and only if $M'$ and $M''$ do.
If $B$ is a thick subcategory of $A$ there is a well defined localization $A/B$, which is again an abelian category. $A/B$ has the same objects as $A$ and a morphism $f:M\to N$ in $A/B$ is an isomorphism if and only if $\ker f$ and $\operatorname{coker} f$ belong to $B$.
Let $T\colon A\to A/B$ be the localization functor. Then $B$ is said to be a localizing subcategory if $B$ is thick and $T$ has a right adjoint. The condition of being localizing can be rephrased only in terms of $A$ and $B$. see Gabriel's thesis above (proposition 4 in chapter III).
Finally, if $M$ is an object of $A$, we denote by $\langle M\rangle$; the smallest localizing subcategory containing $M$.
Now let $X$ be a scheme, $j\colon U \to X$ an open embedding and $i\colon Y\to X$ its closed complement. Then there are a bunch of adjunctions between the categories of quasicoherent sheaves of $U,X,Y$: $i_*\colon \operatorname{QCoh}(Y)\to \operatorname{QCoh}(X)$ has a left adjoint $i^*\colon \operatorname{QCoh}(X)\to \operatorname{QCoh}(Y)$ and a right adjoint $i_!\colon \operatorname{QCoh}(X) \to \operatorname{QCoh}(Y)$. On the other hand, the functor $j^*\colon \operatorname{QCoh}(X)\to \operatorname{QCoh}(U)$ has a left adjoint $j_!\colon \operatorname{QCoh}(U)\to \operatorname{QCoh}(X)$ and a right adjoint $j_*\colon \operatorname{QCoh}(U)\to \operatorname{QCoh}(X)$. This is sometimes called a recollement.
Let's assume that $X$ is Noetherian and let $A = \operatorname{QCoh}(X)$. We have an exact sequence of abelian categories
$$0 \to \operatorname{QCoh}(Y) \to A \to \operatorname{QCoh}(U) \to 0$$
in the sense that the category $\operatorname{QCoh}(Y)$ happens to be a localizing subcategory of $A$ and its quotient is identified with $\operatorname{QCoh}(U)$. The first map in the exact sequence is $i_*$ and the second $j^*$. Moreover, I think that $\operatorname{QCoh}(Y)$ is the smallest localizing subcategory of $\operatorname{QCoh}(X)$ containing $i_*O_Y$. Gabriel proves that there are no more such localizing subcategories, that is closed subschemes of $X$ correspond exactly to localizing subcategories $\langle M\rangle$ generated by a single coherent sheaf (i.e. Noetherian object in $A$). Moreover, irreducible closed subsets (the points in the underlying topological space of $X$) are given by localizing subcategories $\langle I\rangle$ for $I$ an indecomposable injective. We have described the points of $X$ and its closed sets in terms of only the category $A$, so we can recover the underlying topological space of $X$ from $A$.
In particular, an open subscheme $U$ of $X$ gives a complementary closed subscheme $Y$, which is in correspondence with a localizing subcategory $\langle M\rangle$ and, moreover, $\operatorname{QCoh}(U) = A/\langle M\rangle$. So, responding to the queston above, for any $f\colon U\to X$, $U$ is an open subscheme if and only if the kernel of $f^*\colon \operatorname{QCoh}(X) \to \operatorname{QCoh}(U)$ is a localizing subcategory of the form $\langle M\rangle$ for a coherent sheaf $M$.
Regarding the structure sheaf $O_X$ there is an isomorphism between $O_X(U)$ and the ring of endomorphisms of the identity functor on $\operatorname{QCoh}(U)$ (which happens to be $A/\operatorname{QCoh}(Y)$), so the structure sheaf can be recovered only in terms of the category $A$.
Finally, just say that there are other results in the spirit of reconstructing a scheme from some category of sheaves on it. This is the starting point for using such categories of sheaves as a definition of noncommutative scheme. There is more information on this entry in nlab.