First of all, notice that the colimit of a diagram $F:D\rightarrow\mathbf{Top}/X$ can be calculated by taking the colimit $C$ of the functor $$D\rightarrow\mathbf{Top}/X\xrightarrow{U}\mathbf{Top}$$(where $U$ is the "forgetful functor"), then by universal property of colimits, $C$ comes with a unique map $C\rightarrow X$ making it into a colimit for the original diagram $F$.
Next, recall that given a functor $F:C^{op}\times C\rightarrow\mathcal S$ (with $C$ small and $\mathcal S$ cocomplete), the coend is calculated as the coequalizer of the two parallel maps $$\coprod_{d\rightarrow c}F(c,d)\rightrightarrows \coprod_cF(c,c)\tag{*}$$ that when precomposed with the canonical inclusion $F(x,y)\hookrightarrow \coprod_{d\rightarrow c}F(c,d)$ of the $(f:y\rightarrow x)$-factor into the disjoint union, we get the map $$F(x,y)\xrightarrow{F(f,1)}F(y,y)\hookrightarrow\coprod_cF(c,c)$$ and $$F(x,y)\xrightarrow{F(1,f)}F(x,x)\hookrightarrow\coprod_cF(c,c)$$
Now, consider the special case of $F=S\times I:(U,V)\mapsto S(U)\times I(V)$, with $S(U)$ having the discrete topology. As we said above, to calculate the coend of $F$ we can simply consider the induced functor $UF:C^{op}\times C\rightarrow\mathbf{Top}$ which is equal to $$(U,V)\mapsto S(U)\times V$$Then we can describe the coend diagram above in terms of elements: Elements of $\coprod_UF(U,U)$ are represented by pairs $(s,x)$, with $s\in S(U),x\in U$. Elements in $\coprod_{V\rightarrow U}F(U,V)$ are identified by pairs $(s,x)$, with $s\in S(U)$ and $x\in V$, for all inclusions $V\subseteq U$. Then the two maps in (*) acts as follows $$(s,x)\mapsto (s|_V,x),\qquad (s,x)\mapsto (s,x)$$(because the action of $V\subseteq U$ on $S(U)$ is restriction to $V$, while it's action on $I$ is just that of including $x\in V$ as an element of $U$).
The coend is defined as the quotient of $\coprod_UF(U,U)$ by the equivalence relation induced by the two maps $\iota_1,\iota_2$. In particular, notice that for all $x\in V\subseteq U$ and $s\in S(U)$, then $(s,x)\in S(U)\times U$ is identified with $(s|_V,x)\in S(V)\times V$, so if $s\in S(U),t\in S(V)$ are sections such that $s|_W=t|_W$, for some $x\in W\subseteq V\cap U$, then in the coend we have the following chain of equivalencies $$F(U)\times U\ni (s,x)\cong (s|_W,x)=(t|_W,x)\cong (t,x)\in F(V)\times V$$So, if $s,t$ have the same stalk at $x$, then $(s,x),(t,x)$ represent the same element in the coend.
Exercise/Claim: The coend is the quotient of $\coprod_UF(U,U)$ under the equivalence relation $F(U,U)\ni (s,x)\cong (t,y)\in F(V,V)$ if $x=y$ and there is $W\subseteq U\cap V$ such that $s|_W=t|_W$.
This is the connection between the definition using stalks and coends: In both cases, you represent a point of $Y_S$, for a presheaf $S:C^{op}\rightarrow\mathcal Set$, as either a class $s_x\in S_x$(=stalk of $S$ at $x$), or an equivalence class of a pair $(s,x)$. Both $s_x$ and $(s,x)$ are meant to represent the value $s(x)\in Y_S$ of the "section" $s$ on $x$. What you do with stalks is first we identify when two sections $s,t$, defined on some point $x$, should have $s(x)=t(x)$, the answer is if $s,t$ are equal on the stalk $S_x$, so $S_x$ is the candidate for $\pi^{-1}(\{x\})$ (where $\pi:Y_S\rightarrow X$ is the projection), then we put all the fibers together with the appropriate topology and we get $Y_S$ the way Mac Lane does it. In the coend construction, the role of $s(x)$ is fulfilled by the equivalence class of $(s,x)\in F(U)\times U$. This time we first put together $S(U)$-many copies of $U$, for every open $U$, and then identify when $(s,x)$ and $(t,x)$ should represent the same element in $Y_S$. The end result are isomorphic spaces $Y_S\rightarrow X$ having $S$ as sheaf of sections.
This is my first time encountering the construction by coends (which is pretty neat in my opinion), so if there's any error with my interpretation of what is going on, do let me know.
Best Answer
The category of abelian presheaves is locally finitely presentable, and for each abelian presheaf $F$, the functor $F \otimes {-}$ preserves colimits, so we may apply the accessible adjoint functor theorem to obtain a right adjoint $\mathscr{H}om (F, -)$. Representability of the functor $H \mapsto H (S)$ can be used to determine the values of $\mathscr{H}om (F, G)$: $$\mathscr{H}om (F, G) (S) \cong \textrm{Hom} (F \otimes \mathbb{Z} h_S, G)$$ Here $h_S = \mathcal{C} (-, S)$ and $\mathbb{Z} h_S$ is the free abelian presheaf generated by $h_S$.
Actually, your formula is also correct. The point is that you need to verify that $$\textrm{Hom} (F \otimes \mathbb{Z} h_S, G) \cong \textrm{Hom} (F |_{\mathcal{C}_{/ S}}, G |_{\mathcal{C}_{/ S}})$$ which can be done essentially the same way as the Yoneda lemma.