I'll address your questions one by one, referring to the notation in my edit of your question.
(1) In Hartshorne's definition of a closed subscheme, $Y$ is the thing that is actually a scheme. It corresponds directly to Gortz & Wedhorn's $Y$.
Hartshorne's $\iota$ is the inclusion map of $Y$ into $X$. This corresponds to Gortz & Wedhorn's $\iota$.
The reason why Hartshorne talks about equivalence classes is that we need some way of determining whether two closed subschemes of $X$ are essentially the same. For example, suppose $$X = {\rm Spec \ } k[X,Y], \ \ \ \ Y = {\rm Spec \ } k[U], \ \ \ \ Y' = {\rm Spec \ } k[T]$$
and suppose the inclusion morphisms
$$ \iota : Y \to X, \ \ \ \ \ \iota ' : Y' \to X$$
are defined as the scheme morphisms associated to the ring morphisms
$$ k[X,Y] \to k[U], \ \ \ \ \ X \mapsto U, \ \ \ \ \ Y \mapsto 0, $$
and
$$ k[X,Y] \to k[T], \ \ \ \ \ X \mapsto T, \ \ \ \ \ Y \mapsto 0, $$
respectively.
For all intents and purposes, $Y$ embedded by $\iota$ is the same thing as $Y'$ embedded by $\iota '$. There is no difference between $Y$ and $Y'$ apart from my choice of notation. So we should consider them equivalent. Indeed, there exists an isomorphism $\psi : Y' \to Y$ such that $\psi \circ \iota = \iota '$: this $\psi$ is the morphism of schemes associated to the ring morphism sending $U \mapsto T$. So $Y$ and $Y'$ with their respective inclusion maps are considered equivalent by Hartshorne's definition.
The scheme structures are well-defined on equivalence classes because if $\iota : Y \to X$ and $\iota ' : Y ' \to X$ are two equivalent closed subschemes with their respective immersions, then $Y$ and $Y'$ are isomorphic as schemes (via $\psi$) by definition, and therefore, they have the same scheme structures.
(2) Suppose the scheme $Y$ together with the embedding $\iota : Y \to X$ is a closed subscheme by Hartshorne's definition. How can we see that the quasi-coherent sheaf $\iota_\star \mathcal O_Y$ is of the form $\mathcal O_X / \mathcal I$ for some quasi-coherent sheaf $\mathcal I$ that is a sheaf of ideals of $\mathcal O_X$? We would like to do this in order to match up with Gortz & Wedhorn's definition.
Well, we have a surjective morphism of sheaves:
$$ \mathcal O_X \twoheadrightarrow \iota_\star \mathcal O_Y$$
(This morphism of sheaves is the $\iota^{\#}$ in Hartshorne's definition.)
Since the category of quasi-coherent sheaves on $X$ is an abelian category, there exists a quasi-coherent sheaf $\mathcal I$ that acts as the kernel for the above sheaf morphism, giving a short exact sequence:
$$ 0 \to \mathcal I \hookrightarrow \mathcal O_X \twoheadrightarrow \iota_\star \mathcal O_Y \to 0$$
But then, $\mathcal I $ is a sub-$\mathcal O_X$-module of $\mathcal O_X$, which is the same as saying that $\mathcal I$ is a sheaf of ideals of $\mathcal O_X$, and furthermore, $\iota_\star \mathcal O_Y \cong \mathcal O_X / \mathcal I$ as sheaves, so we have constructed everything that is required in Gortz & Wedhorn's definition.
Note that we are working in the category of quasi-coherent sheaves on $X$, and not in the category of rings. Your comment about the category of rings not being an additive category is unimportant. (Here is a local version of my statement: on an affine scheme, specifying a quasi-coherent sheaf is equivalent to specifying a module over a fixed ring. The category of modules over a fixed ring is certainly an abelian category, even if the category of rings is not.)
It should also possible to go backwards from Gortz & Wedhorn's definition to Hartshorne's definition. The annoying thing is that Hartshorne wants $\iota : Y \to X$ to be a morphism of schemes, whereas the $\iota : Y \to X$ provided by Gortz & Wedhorn is only an inclusion of topological spaces. So we need to specify a sheaf morphism $\iota^\# : \mathcal O_X \to \iota_\star \mathcal O_Y$ such that $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ defines a genuine morphism of schemes. It is clear that we ought to define $\iota^\#$ to be the composition
$ \mathcal O_X \twoheadrightarrow \mathcal O_X / \mathcal I \cong \mathcal O_Y$
in order to ensure that the current construction really is the inverse of the previous one. But unless I'm missing something obvious, I can't any simple way of showing that $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ is a morphism of schemes without doing a lot of work. Personally, I would start by arguing that the only way that $\iota_\star \mathcal O_Y$ can be isomorphic to $\mathcal O_X / \mathcal I $ is if $Y$ is the support of the sheaf $\mathcal O_X / \mathcal I$. Since the question is of a local nature, I would now focus my attention on an open affine $U \cong {\rm Spec \ } A \subset X$. Writing $\mathcal I|_U$ as $\widetilde{I}$ for some ideal $I \subset A$, I would show that the support of $\mathcal O_X / I$ is the set $V(I) \subset {\rm Spec \ } A$, which is homeomorphic to ${\rm Spec \ }(A/I)$ as a topological space. Finally, by examining sections on basic open sets and restriction maps between basic open sets, I would try to show that $(Y, \mathcal O_Y)$ has the structure of ${\rm Spec \ } (A / I)$ as a scheme, and that the map $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ is the morphism of schemes associated to the natural ring morphism $A \twoheadrightarrow A / I$.
[Apologies for earlier edits of this post where I misread your statement of Gortz & Wedhorn!]
(3) I'm afraid that your hypothesis that ${\rm Spec}(A / I)$ and ${\rm Spec}(A / I')$ are equivalent when $\sqrt{I} = \sqrt{I'}$ is not true. For example, consider:
$$ X = {\rm Spec \ } k[T], \ \ \ \ \ Y = {\rm Spec \ } k[T] / (T), \ \ \ \ \ \ \ Y' = {\rm Spec \ } k[T] / (T^2).$$
$Y$ can be thought of as a closed subvariety of $X$: it is the point at the origin. But $Y'$ is a totally different thing - it isn't a variety at all. $Y'$ should be thought of as a "double point" at the origin.
From Hartshorne's perspective, $Y$ and $Y'$ cannot possibly be isomorphic because $k[T]/(T)$ is a reduced ring whereas $k[T]/(T^2)$ is non-reduced, so it's impossible to construct a scheme isomorphism between $Y$ and $Y'$.
And how are $Y$ and $Y'$ different from Gortz & Wedhorn's point of view? After all, the underlying topological spaces are the same, and the inclusion maps are also the same when viewed as continuous maps between topological spaces. The difference between $Y$ and $Y'$ is that their structures sheaves are different. $\iota_\star \mathcal O_Y$ is isomorphic to $\mathcal O_X / \mathcal I$ with $\mathcal I = \widetilde{(T)}$, whereas $\iota_\star \mathcal O_{Y'}$ is isomorphic to $\mathcal O_X / \mathcal I'$ with $\mathcal I' = \widetilde{(T^2)}$.
Best Answer
You have a misconception here: $W$ is not just some arbitrary open set! Let's start by reproducing the text:
$\Delta(X)$ is a closed subscheme of a specific open subscheme $W\subset X\times_YX$. From here, we have a closed immersion $i:\Delta(X)\hookrightarrow W$, which gives us that the ideal sheaf $\mathscr{I}$ of $\Delta(X)$ inside $W$ is exactly the kernel of the morphism $i^\sharp: \mathcal{O}_W\to i_*\mathcal{O}_{\Delta(X)}$. (This part of your interpretation is correct, and is exactly the definition of a closed immersion.)
Here are some examples:
In the case when $X\to Y$ is separated, then $\Delta(X)$ is honestly closed inside $X\times_Y X$. For instance, if $X=\Bbb A^1_R=\operatorname{Spec} R[x]$ and $Y=\operatorname{Spec} R$, then $X\times_Y X=\operatorname{Spec} R[x_1,x_2]$, $W=X\times_YX$ and $\Delta(X)$ is the closed subscheme cut out by the ideal $(x_1-x_2)$, which is the kernel of the map $R[x_1,x_2]\to R[x]$ given by sending $x_1\mapsto x$ and $x_2\mapsto x$. (Here, I'm using the equivalence between quasicoherent sheaves on affine schemes and modules over the ring of global sections to work with modules instead of sheaves. If you don't like it, hit everything with the tilde/associated sheaf functor.)
One can generalize (1) to the case of $X,Y$ affine, represented by $\operatorname{Spec} A$ and $\operatorname{Spec} B$, respectively. Since affine morphisms are separated, we can again take $W=X\times_Y X$ and calculate that the sheaf of ideals of $\Delta(X)$ is the sheaf associated to the module generated by $1\otimes_B a - a\otimes_B 1$ for all $a\in A$, as the structure sheaf of $X\times_YX$ is the sheaf associated to the module $A\otimes_BA$, which maps to $A$ by the obvious multiplication map sending $x\otimes_B y\mapsto xy$.
In the case where $X\to Y$ is not separated, we actually need to pick $W$ in an interesting way. Consider the easiest example: let $Y=\operatorname{Spec} k$ and $X$ the affine line over $k$ with two origins $0_1,0_2$. Then the two closed points of $X\times_Y X$ given by $(0_1,0_2)$ and $(0_2,0_1)$ are in $\overline{\Delta(X)}$ but not in $\Delta(X)$, so we can construct $W$ to kick them out: let $W=(X\times_Y X) \setminus \{(0_1,0_2),(0_2,0_1)\}$. Now $\Delta$ is an honestly closed subscheme of $W$, and we can then compute the kernel of the map of structure sheaves affine-locally as in case 2 and patch together.