First of all, as I've said in the comments, I don't understand what you mean by "setting $M$ to be the modulo of $\operatorname{im}\alpha$", so I can't help you there.
Regarding your last paragraph, it doesn't work. In Abelian category $gf = 0\implies g = 0$ if and only if $f$ is epimorphism. For all we know, your $\alpha$ could be zero map.
Now, to the proof.
- $F$ is mono:
Let $Fs = 0$. Then $s\circ \beta = 0$ and since $\beta$ is epimorphism, $s = 0$.
- $G\circ F = 0$ (equivalently, $\operatorname{im}F\subseteq \ker G$):
$(G\circ F)(s) = s\circ \beta \circ \alpha = 0$ since $\beta\circ\alpha = 0$.
- $ \ker G\subseteq\operatorname{im}F$:
Let $Gs = 0$, i.e. $s\circ\alpha = 0$. Note that $\beta$ is cokernel of $\alpha$, so by the universal property of cokernel, there exists $s'$ such that $s'\circ\beta= s$. Thus $s\in\operatorname{im}F$.
EDIT: Let me elaborate on the last paragraph. Let's say we work with modules or Abelian groups, it works the same in any Abelian category (just without elements).
Let's start by noting that $B/\ker\beta\cong \operatorname{im}\beta = C$ by the first isomorphism theorem. Write $\varphi\colon C\to B/\ker\beta$ for the inverse of the map $b+\ker\beta\mapsto \beta(b)$. Immediately we have that the composition $\varphi\circ \beta$ is the canonical epimorphism $B \twoheadrightarrow B/\ker\beta .$ Now, since $\ker\beta=\operatorname{im}\alpha$ (it's strict equality, not just isomorphism), we can write $B/\operatorname{im}\alpha$ instead of $B/\ker\beta$.
Now, let $s\colon B\to M$ be such that $s\circ\alpha = 0$. I claim that there exists unique map $t\colon B/\operatorname{im}\alpha\to M$ to make the following diagram commute:
$\require{AMScd}$
\begin{CD}
A @>\alpha>> B @>\beta>> C @>>> 0\\
@| @| @V\varphi VV \\
A @>\alpha>> B @>\small\text{canon. epi}>> B/\operatorname{im}\alpha @>>> 0\\
@. @VsVV @VtVV \\
@. M @= M
\end{CD}
Define $t(b+\operatorname{im}\alpha) = s(b)$. I will leave the verification that this is well-defined to you. Uniqueness of such a map is obvious from the definition.
Finally, define $s'\colon C\to M$ by setting $s' = t\circ \varphi$.
All in all, what we have just proved is the following:
Let $A\stackrel{\alpha}{\to} B \stackrel{\beta}{\to} C \to 0$ be an
exact sequence. For every $s\colon B\to M$ such that $s\circ\alpha =
0$ there exists unique $s'\colon C\to M$ such that $s'\circ\beta = s$.
This is called the universal property of cokernel. In this case $\beta$ is the cokernel of $\alpha$. More precisely, the pair $(C,\beta)$ is the cokernel, but we often omit mentioning the object since it is understood from the context.
Thus, cokernel is not just an object, it is a pair of an object and a map onto it. Canonical choice for cokernel is $B \twoheadrightarrow B/\operatorname{im}\alpha $ where the arrow is the canonical epimorphism. But, in our example, we used that this canonical cokernel is isomorphic to $C$ since we really wanted a map from $C$, not from $B/\operatorname{im}\alpha$.
But, it is not enough that $B/\operatorname{im}\alpha \cong C$ to say that $C$ is cokernel, the maps onto $C$ and $B/\operatorname{im}\alpha$ must commute with the isomorphism, as you can see from the diagram. The point is, objects themselves don't matter much on their own, it's the maps that do.
Generally looks good, though as Berci says in the comments you should check that $\phi$ doesn't depend on your choices.
However, I write an answer, because I would suggest an alternative method of proof for
$\newcommand\im{\operatorname{im}}\ker\alpha^*=\im\beta^*$.
The key is to notice that $C\cong B/\im\alpha$, and $\beta : B\to C$ is the quotient map.
Therefore you can use the universal property of the quotient, which is that maps $\psi : B\to M$ such that $\psi(\im\alpha)=0$ are in one-to-one correspondence with maps $\tilde{\psi} : C\to M$, and the correspondence is given by $\psi = \tilde{\psi}\circ \beta$.
Then $\psi(\im\alpha)=0$ if and only if $\alpha^*\psi = \psi\circ \alpha =0$ if and only if $\psi\in \ker\alpha^*$. Thus $\psi\in\ker\alpha^*$ if and only if
$\psi=\tilde{\psi}\circ \beta$ for some $\tilde{\psi}\in \operatorname{Hom}_R(C,M)$, i.e., $\psi \in \ker\alpha^*$ if and only if $\psi \in \im\beta^*$, as desired.
Best Answer
First of all, $g\circ\beta=0$ means for all $y\in M$, $g(\beta(y))=0$ (not only for some).
So let $y\in M$. Since $\beta(y)\in\ker(g)=im(f)$, there exists $x\in A$ such that $\beta(y)=f(x)$. The crucial point that you're missing is this one: since $f$ is injective by assumption, this $x$ is unique!! (if $\beta(y)=f(x_1)=f(x_2)$, then $x_1=x_2$...)
Hence, we may denote by $\alpha(y)$ this $x$, and we get a map $\alpha:M\to A$ such that $\beta(y)=f(\alpha(y))$ for all $y\in M$. Thus, $\beta=f\circ \alpha$.
Now it remains to prove that $\alpha$ is $R$-linear. By definition, for $y\in M$, $\alpha(y)$ is the unique element of $M$ such that $\beta(y)=f(\alpha(y))$.
But for all $y_1,y_2\in M$ and all $r\in R$, we have $\beta(y_1+ry_2)=\beta(y_1)+r\beta(y_2)=f(\alpha(y_1))+r f(\alpha(y_2))=f(\alpha(y_1)+r\alpha(y_2))$. But uniqueness above, $\alpha(y_1+ry_2)=\alpha(y_1)+r\alpha(y_2)$, and we are done.
Since $\alpha$ is $R$-linear we have $\beta=f\circ\alpha=f_*(\alpha)$.