I have been confused by an example from Dolgachev: Derived Categories, which aims to show that the cokernel of a morphism between two presheaves $h_A$ and $h_B$ may not be representable, even when $\cal C=\mathbf{Ab}$ where kernels and cokernels are the usual ones.
The example is that if we take the multiplication by $n$, $[n]: \mathbb Z\to Z$ in $\cal Ab$ and $u=h([n]):h_{\mathbb Z}\to h_{\mathbb Z}$, then we get
$$\mathrm{coker}(u)(\mathbb Z/n\mathbb Z)=\mathrm{coker}(h_{\mathbb Z}(\mathbb Z/n\mathbb Z)\to h_{\mathbb Z}(\mathbb Z/n\mathbb Z))=\mathrm{coker}(0\to 0)=\{0\},$$
because $\mathrm{Hom}(\mathbb Z/n\mathbb Z, \mathbb Z)=\{0\}$. If $\mathrm{coker}(u)$ is representable, then $\mathrm{coker}([n])$ is clearly a representing object. However,
$$h_{\mathrm{coker}([n])}(\mathbb Z/n\mathbb Z)=h_{\mathbb Z/n\mathbb Z}(\mathbb Z/n\mathbb Z)=\mathrm{Hom}(\mathbb Z/n\mathbb Z,\mathbb Z/n\mathbb Z)\neq \{0\}.$$
And this shows that $\mathrm{coker}(u)$ is not representable.
My question is that: why shall the object $\mathrm{coker}([n])$ be suspected as a representing object "automatically"? The example above is mainly trying to show that this object cannot be a representing object and then conclude that $\mathrm{coker}(u)$ is not representable by ANY object in $\cal C$, but why is $\mathrm{coker}([n])$ the ONLY possible choice to represent $\mathrm{coker}(u)$?
So some background goes here:
For a small category $\cal C$, denote by $\hat{\cal C}={\cal Sets}^{{\cal C}^{\mathrm{op}}}$ the category of presheaves on $\cal C$. Any object $A$ in $\cal C$ defines a presheaf, i.e. the Hom-functor:
$$h_A=\mathrm{Hom}(-,A): (\phi: X\to Y)\to (\mathrm{Mor}(Y,A)\overset{\phi^\circ}{\rightarrow} \mathrm{Mor}(X,A)).$$A presheaf $F\in\hat{C}$ is called representable if it is isomorphic to a functor of the form $h_S$ for some representing object $S\in\mathrm{ob}(\cal C)$. The representing object of a representable presheaf is defined uniquely up to isomorphism.
Let $\hat{\cal C}^\mathrm{ab}$ be the category of abelian presheaves on $\cal C$, i.e. the category of contravariant functors from $\cal C$ to $\mathbf {Ab}$. If $\cal C$ is a $\mathbb Z$-category then the Yoneda functor is from $\cal C$ to $\hat{\cal C}^\mathrm{ab}$.
For any morphism $u:F\to G$ of abelian presheaves we define the kernel of $u$ by
$$\mathrm{ker}(u)(A)=\mathrm{ker}(F(A)\to G(A)),$$
which can be characterised by the universal property: there is a morphism $\mathrm{ker}(u)\to F$ s.t. the composition $\mathrm{ker}(u)\to F\to G$ is the zero morphism, and any morphism $K\to F$ with this property factors through $\mathrm{ker}(u)\to F$.By the Yoneda Lemma, the kernel $\mathrm{ker(X\to Y)}$ satisfies the universal property from above, so this can be taken as the equivalent definition.
Similarly for a pre-additive category, by
$$\mathrm{coker}(u)=\mathrm{ker}(u^{\mathrm{op}})^{\mathrm{op}},$$
we can define the cokernel of a morphism in $\cal C$. And $\hat{\cal C}^{\mathrm{op}}$ has cokernels defined by
$$\mathrm{coker}(F\to G)(S)=\mathrm{coker}(F(S)\to G(S)).$$
Best Answer
I do not want to go into full detail, since this might get messy, regarding notation. However, what is actually happening here is that you embed via yoneda your objects $A,B\in $\mathcal{C}$ $ as $h_A$ and $h_B$, now we know that the category of presheafs in abelian groups is abelian, hence has cokernels. However, your category $\mathcal{C}$ might not have them.
Now you want to show that it really does not have them, which as you already said can be tricky, as you do not know how the cokernel would look like. But you are now living in a bigger categroy that has cokernels (presheaves in abelian groups) and so you know how the cokernels look like (up to unique isomorphism) in there. and so you can pick 2 objects, like in your question, whose cokernel you know, and show that that one is not representable.
Please forgive me, wanted to comment this, but the space was too small. Hope you still accept it as an answer, although it lacks all details. But it should still answer the question, why one may assume that the cokernel has a certain form.