If $X=Spec(A)$, we have $X_ \text {red}=Spec(A_\text {red})$, where $A_{\text {red}}=A/\text{Nil(A)}$ so that:
$\Gamma(X,\mathcal O_X)=A$
but
$\Gamma(X_{\text {red}},\mathcal O_{X_ \text {red}})=A_{\text {red}}=A/\text{Nil(A)}$
No contradiction with Hartshorne's 2.2.2.(c)!
Edit: some details
Here are some statements which might help shed light on this subtle question.
a) Given a scheme $X$ we associate to it the quasi-coherent sheaf of ideal $\mathcal N\subset \mathcal O_X$ defined for an arbitrary open subset $U\subset X$ by $$\mathcal N(U)=\{f\in \mathcal O_X(U)\mid \forall x\in \mathcal O_{X,x },\; f_x \in \text {Nil}(\mathcal O_{X,x }) \}$$ b) The scheme $X_{\text {red}}$ has structure sheaf $\mathcal O_{X_{\text {red}}}=\mathcal O_X/\mathcal N$
c) For any affine subset $U=\text {Spec} (A)\subset X$, we have $ \text {Nil}(\Gamma(U,\mathcal O_X))=\mathcal N(U)=\text {Nil}(A)$
d) For any affine subset $U=\text {Spec} (A)\subset X$, we have $\mathcal O_{X_{\text {red}}}(U)=A_{\text {red}}=A/{\text {Nil}} (A)$
e) For a general open subset $U\subset X$ , we have $ \text {Nil}(\Gamma(U,\mathcal O_X))\subset \mathcal N(U)$
but the inclusion may be strict for non-affine $U$:
Let $X_m=\text {Spec}(\mathbb C[T]/T^m)=\text {Spec}(\mathbb C[\epsilon _m])$ and $X=\bigsqcup X_m$ (a non-affine scheme).
Then $\Gamma(X,\mathcal O_X)=\prod \Gamma(X_m,\mathcal O_{X_m})=\prod \mathbb C[\epsilon _m]$ and for $\epsilon=(\epsilon_1,\epsilon_2,\cdots)$ we have $\epsilon \notin \text {Nil}(\Gamma(X,\mathcal O_X))$ although $\epsilon \in \mathcal N(X)$.
There's a lot of confusion in your post: as is, your proposed morphism of sheaves does not make any sense. The map you consider should not be the embedding $\pi \colon D(f) \to \mathrm{Spec}(A)$, but rather the embedding $\mathrm{Spec}(\alpha) \colon \mathrm{Spec}(A_{f}) \to \mathrm{Spec}(A)$ induced by the canonical localization map $\alpha \colon A \to A_{f}$. As you note, $\pi := \mathrm{Spec}(\alpha)$ is an open embedding whose image is $D(f)$, so we may view it as an isomorphism of topological spaces $\mathrm{Spec}(A_{f}) \to D(f)$.
Moving to sheaves, let me recall what the sheaf $\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)}$ is. For any open set $U \subset D(f)$, $U$ is likewise an open set of $\mathrm{Spec}(A)$, and by definition we have $\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)}(U) = \mathcal{O}_{\mathrm{Spec}(A)}(U)$. The key, then, is understanding which distinguished opens of $\mathrm{Spec}(A)$ are contained in $D(f)$ - more on this shortly. Moreover, the map $\pi$ comes with an associated morphism of sheaves $\mathcal{O}_{\mathrm{Spec}(A)} \to \pi_{\ast}\mathcal{O}_{\mathrm{Spec}(A_{f})}$, which on global sections is $\alpha$ and on distiniguished opens is the (induced) localization map. The corresponding morphism of sheaves $\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)} \to \pi_{\ast}\mathcal{O}_{\mathrm{Spec}(A_{f})}$ is induced by $\mathcal{O}_{\mathrm{Spec}(A)} \to \pi_{\ast}\mathcal{O}_{\mathrm{Spec}(A_{f})}$ in the obvious way; on global sections, it is the identity map $A_{f} \to A_{f}$, since $\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)}(D(f)) = \mathcal{O}_{\mathrm{Spec}(A)}(D(f)) = A_{f}$, and $\pi^{-1}(D(f)) = \mathrm{Spec}(A_{f})$.
All that remains is to understand why $\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)} \to \pi_{\ast}\mathcal{O}_{\mathrm{Spec}(A_{f})}$ is an isomorphism of sheaves on $D(f)$. It suffices to check this on a basis for the topology on $D(f)$, which is given by the distinguished opens of $\mathrm{Spec}(A)$ contained in $D(f)$. With the above details settled, here is a guide to the approach, which I leave to you.
(1) First, show that we have a containment of distinguished opens $D(g) \subset D(f)$ if and only if $f$ is a unit of $A_{g}$. (This is exercise 3.5F of Vakil - very much worth doing, if you haven't done so yet.)
(2) Next, show that $\pi^{-1}(D(g)) = D(\alpha(g)) = D(g/1)$ for any $g \in A$. (There is nothing special about $\pi$ here, to be clear: for any morphism of rings $u \colon A \to B$ and any $g \in A$, one has $\mathrm{Spec}(u)^{-1}(D(g)) = D(u(g))$.)
(3) Finally, we put things together. Let $D(g)$ be a distinguished open of $\mathrm{Spec}(A)$ which is contained in $D(f)$, which by (1) ensures that $f$ is a unit in $A_{g}$. We have
$$\pi_{\ast}\mathcal{O}_{\mathrm{Spec}(A_{f})}(D(g)) = \mathcal{O}_{\mathrm{Spec}(A_{f})}(D(\pi(g))) = (A_{f})_{g/1}$$
and
$$\mathcal{O}_{\mathrm{Spec}(A)}|_{D(f)}(D(g)) = \mathcal{O}_{\mathrm{Spec}(A)}(D(g)) = A_{g}$$.
The map $A_{g} \to (A_{f})_{g/1}$ is the universal map induced by $\alpha \colon A \to A_{f}$. Your task is to show that this map $A_{g} \to (A_{f})_{g/1}$ is an isomorphism, which I leave to you. (I would use the universal property of localization to get a map $(A_{f})_{g/1} \to A_{g}$. You will use that $f$ is invertible in $A_{g}$ to get a map $A_{f} \to A_{g}$ first.)
Best Answer
Careful. You have shown that a the sections of a presheaf $\mathscr G$ over a base for the topology on some space $X$ are isomorphic to those of a sheaf $\mathscr F$ ($\mathscr G$, $\mathscr F$ are assumed to be presheaves on $X$). This is not enough to conclude abstractly that $\mathscr G$ is a sheaf isomorphic to $\mathscr F$; such a thing is clearly not true in general. But it is true that the sheaf associated to $\mathscr G$ is isomorphic to $\mathscr F$. This is exactly the definition of the structure sheaf of the reduced scheme associated to $X$ that Hartshorne uses.
I think the proof is clear.