I'll address your questions one by one, referring to the notation in my edit of your question.
(1) In Hartshorne's definition of a closed subscheme, $Y$ is the thing that is actually a scheme. It corresponds directly to Gortz & Wedhorn's $Y$.
Hartshorne's $\iota$ is the inclusion map of $Y$ into $X$. This corresponds to Gortz & Wedhorn's $\iota$.
The reason why Hartshorne talks about equivalence classes is that we need some way of determining whether two closed subschemes of $X$ are essentially the same. For example, suppose $$X = {\rm Spec \ } k[X,Y], \ \ \ \ Y = {\rm Spec \ } k[U], \ \ \ \ Y' = {\rm Spec \ } k[T]$$
and suppose the inclusion morphisms
$$ \iota : Y \to X, \ \ \ \ \ \iota ' : Y' \to X$$
are defined as the scheme morphisms associated to the ring morphisms
$$ k[X,Y] \to k[U], \ \ \ \ \ X \mapsto U, \ \ \ \ \ Y \mapsto 0, $$
and
$$ k[X,Y] \to k[T], \ \ \ \ \ X \mapsto T, \ \ \ \ \ Y \mapsto 0, $$
respectively.
For all intents and purposes, $Y$ embedded by $\iota$ is the same thing as $Y'$ embedded by $\iota '$. There is no difference between $Y$ and $Y'$ apart from my choice of notation. So we should consider them equivalent. Indeed, there exists an isomorphism $\psi : Y' \to Y$ such that $\psi \circ \iota = \iota '$: this $\psi$ is the morphism of schemes associated to the ring morphism sending $U \mapsto T$. So $Y$ and $Y'$ with their respective inclusion maps are considered equivalent by Hartshorne's definition.
The scheme structures are well-defined on equivalence classes because if $\iota : Y \to X$ and $\iota ' : Y ' \to X$ are two equivalent closed subschemes with their respective immersions, then $Y$ and $Y'$ are isomorphic as schemes (via $\psi$) by definition, and therefore, they have the same scheme structures.
(2) Suppose the scheme $Y$ together with the embedding $\iota : Y \to X$ is a closed subscheme by Hartshorne's definition. How can we see that the quasi-coherent sheaf $\iota_\star \mathcal O_Y$ is of the form $\mathcal O_X / \mathcal I$ for some quasi-coherent sheaf $\mathcal I$ that is a sheaf of ideals of $\mathcal O_X$? We would like to do this in order to match up with Gortz & Wedhorn's definition.
Well, we have a surjective morphism of sheaves:
$$ \mathcal O_X \twoheadrightarrow \iota_\star \mathcal O_Y$$
(This morphism of sheaves is the $\iota^{\#}$ in Hartshorne's definition.)
Since the category of quasi-coherent sheaves on $X$ is an abelian category, there exists a quasi-coherent sheaf $\mathcal I$ that acts as the kernel for the above sheaf morphism, giving a short exact sequence:
$$ 0 \to \mathcal I \hookrightarrow \mathcal O_X \twoheadrightarrow \iota_\star \mathcal O_Y \to 0$$
But then, $\mathcal I $ is a sub-$\mathcal O_X$-module of $\mathcal O_X$, which is the same as saying that $\mathcal I$ is a sheaf of ideals of $\mathcal O_X$, and furthermore, $\iota_\star \mathcal O_Y \cong \mathcal O_X / \mathcal I$ as sheaves, so we have constructed everything that is required in Gortz & Wedhorn's definition.
Note that we are working in the category of quasi-coherent sheaves on $X$, and not in the category of rings. Your comment about the category of rings not being an additive category is unimportant. (Here is a local version of my statement: on an affine scheme, specifying a quasi-coherent sheaf is equivalent to specifying a module over a fixed ring. The category of modules over a fixed ring is certainly an abelian category, even if the category of rings is not.)
It should also possible to go backwards from Gortz & Wedhorn's definition to Hartshorne's definition. The annoying thing is that Hartshorne wants $\iota : Y \to X$ to be a morphism of schemes, whereas the $\iota : Y \to X$ provided by Gortz & Wedhorn is only an inclusion of topological spaces. So we need to specify a sheaf morphism $\iota^\# : \mathcal O_X \to \iota_\star \mathcal O_Y$ such that $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ defines a genuine morphism of schemes. It is clear that we ought to define $\iota^\#$ to be the composition
$ \mathcal O_X \twoheadrightarrow \mathcal O_X / \mathcal I \cong \mathcal O_Y$
in order to ensure that the current construction really is the inverse of the previous one. But unless I'm missing something obvious, I can't any simple way of showing that $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ is a morphism of schemes without doing a lot of work. Personally, I would start by arguing that the only way that $\iota_\star \mathcal O_Y$ can be isomorphic to $\mathcal O_X / \mathcal I $ is if $Y$ is the support of the sheaf $\mathcal O_X / \mathcal I$. Since the question is of a local nature, I would now focus my attention on an open affine $U \cong {\rm Spec \ } A \subset X$. Writing $\mathcal I|_U$ as $\widetilde{I}$ for some ideal $I \subset A$, I would show that the support of $\mathcal O_X / I$ is the set $V(I) \subset {\rm Spec \ } A$, which is homeomorphic to ${\rm Spec \ }(A/I)$ as a topological space. Finally, by examining sections on basic open sets and restriction maps between basic open sets, I would try to show that $(Y, \mathcal O_Y)$ has the structure of ${\rm Spec \ } (A / I)$ as a scheme, and that the map $(\iota, \iota^\#) : (Y, \mathcal O_Y) \to (X, \mathcal O_X)$ is the morphism of schemes associated to the natural ring morphism $A \twoheadrightarrow A / I$.
[Apologies for earlier edits of this post where I misread your statement of Gortz & Wedhorn!]
(3) I'm afraid that your hypothesis that ${\rm Spec}(A / I)$ and ${\rm Spec}(A / I')$ are equivalent when $\sqrt{I} = \sqrt{I'}$ is not true. For example, consider:
$$ X = {\rm Spec \ } k[T], \ \ \ \ \ Y = {\rm Spec \ } k[T] / (T), \ \ \ \ \ \ \ Y' = {\rm Spec \ } k[T] / (T^2).$$
$Y$ can be thought of as a closed subvariety of $X$: it is the point at the origin. But $Y'$ is a totally different thing - it isn't a variety at all. $Y'$ should be thought of as a "double point" at the origin.
From Hartshorne's perspective, $Y$ and $Y'$ cannot possibly be isomorphic because $k[T]/(T)$ is a reduced ring whereas $k[T]/(T^2)$ is non-reduced, so it's impossible to construct a scheme isomorphism between $Y$ and $Y'$.
And how are $Y$ and $Y'$ different from Gortz & Wedhorn's point of view? After all, the underlying topological spaces are the same, and the inclusion maps are also the same when viewed as continuous maps between topological spaces. The difference between $Y$ and $Y'$ is that their structures sheaves are different. $\iota_\star \mathcal O_Y$ is isomorphic to $\mathcal O_X / \mathcal I$ with $\mathcal I = \widetilde{(T)}$, whereas $\iota_\star \mathcal O_{Y'}$ is isomorphic to $\mathcal O_X / \mathcal I'$ with $\mathcal I' = \widetilde{(T^2)}$.
Best Answer
The definition of open subscheme in Hartshorne is so bad that the first sentence he writes after it is false!
That sentence is: Note that every open subset of a scheme carries a unique structure of open subscheme (Ex.2.2).
This is false: indeed, replacing $\mathcal O_{X|U}$ by an isomorphic sheaf you will get another scheme (albeit isomorphic, of course) and the claimed unicity does not hold..
And you really get other schemes because a scheme is a pair formed by a topological space and a sheaf of rings, not an isomorphism class of sheaves of rings.
So that the collection of subscheme structures on an open subset $U$ of a scheme $X$ would not even be a set with Hartshorne's definition ...
Of course in EGA, Görz-Wedhorn, Qing Liu,... you will find the reasonable definition that an open subscheme of $X$ is an open subset $U\subset X$ endowed with the restricted sheaf $\mathcal O_{X|U}$ , and not some arbitrary sheaf isomorphic to it. There is of course a canonical morphism of schemes $j:U\to X$
Edit: Open immersions Once you have the correct notion of open subscheme, you have no choice for defining an open immersion $Y\to X$. The natural idea is that you want it to be an isomorphism $g:Y \stackrel {\sim}{\to} U $ where $U\subset X$ is an open subscheme. However this has the wrong target, so you just compose it with the canonical morphism $j:U\to X$ mentioned above and you get the required immersion $f=j\circ g:Y\to X$, just as Wikipedia and most other references say.
And, quite satisfactorily, $j:U\to X$ itself is then an open immersion ( take $g=id_U$).
A criterion for being an open immersion: In practice to prove that a morphism $f:Y\to X$ is an open immersion, it suffices to check that:
a) $f$ induces a homeomorphism of $Y$ onto an open subset $U$ of $X$.
b) For all $y\in Y$ the local morphism $f^\ast_y: \mathcal O_{X,f(y)} \to \mathcal O_{Y,y}$ is an isomorphism of local rings.
NB Needless to say, I have an immense admiration for Hartshorne: I'm criticizing a definition in a book, and certainly not that great, friendly mathematician!