Well, your proof is okay. Let me suggest a slightly different way of looking at it:
Consider a sequence
$$ 0\; \xrightarrow{\phantom{ij}} \; A \; \xrightarrow{i\phantom{j}}\; B\; \xrightarrow{j\phantom{i}} \; C$$
and look at
$$ 0\; \xrightarrow{\phantom{j_{\ast}}}
\operatorname{Hom}{(M,A)}\; \xrightarrow{i_{\ast}}\operatorname{Hom}{(M,B)} \;\xrightarrow{j_{\ast}}\;
\operatorname{Hom}{(M,C)},$$
where I write $i_{\ast} = \operatorname{Hom}(M,i)$ and $j_{\ast} = \operatorname{Hom}(M,j)$.
Saying that the first sequence is exact amounts to saying $i = \ker{j}$, that is $ji = 0$ and $i$ has the universal property as depicted in the first diagram below: If $g: M \to B$ is such that $jg = 0$ then there exists a unique $f: M \to A$ such that $if = g$. In other words if $j_\ast g = 0$ then $g = i_{\ast}f$, or yet again $\operatorname{Ker}{j_\ast} \subset \operatorname{Im}{i_{\ast}}$ and $i_{\ast}$ is injective.
On the other hand, the second diagram says: if $g: M \to B$ is of the form $g = if = i_{\ast}f$ then $j_{\ast}g = 0$ (because $j_\ast g = j_{\ast}i_{\ast} f = (ji)_{\ast}f = 0f = 0$). In other words, $\operatorname{Im}{i_{\ast}} \subset \operatorname{Ker}{j_{\ast}}$.
Summing up, we have shown that for all $M$ the sequence
$$ 0\; \xrightarrow{\phantom{j_{\ast}}}
\operatorname{Hom}{(M,A)}\; \xrightarrow{i_{\ast}}\operatorname{Hom}{(M,B)} \;\xrightarrow{j_{\ast}}\;
\operatorname{Hom}{(M,C)}$$
is exact both at $\operatorname{Hom}{(M,A)}$ ($i_{\ast}$ is injective) and at $\operatorname{Hom}{(M,B)}$ ($\operatorname{Im}{i_{\ast}} = \operatorname{Ker}{j_{\ast}}$)—you seem to have forgotten about the first point here.
Added: As witnessed by the argument above, left exactness of $\operatorname{Hom}$ is essentially the definition of left exactness in the abelian category of $R$-modules. As the comments try to point out, the importance of this fact cannot be overemphasized.
I would like to add two further points:
A functor $F$ is left exact in your definition if and only if $0 \to F(A) \to F(B) \to F(C)$ is left exact for every short exact sequence $0 \to A \to B \to C \to 0$.
Indeed, in a left exact sequence $0 \to A \to B \to C$, we may factor $j: B \to C$ over its image as $B \twoheadrightarrow \operatorname{Im}{j} \rightarrowtail C$ and obtain two exact sequences $$0 \to A \to B \to \operatorname{Im}{j} \to 0 \qquad \text{and} \qquad 0 \to \operatorname{Im}{j} \to C \to \operatorname{Coker}{j} \to 0.$$ Applying $F$ to these two exact sequences, we obtain the left exact sequences $$0 \to F(A) \to F(B) \to F(\operatorname{Im}{j}) \qquad \text{and} \qquad 0 \to F(\operatorname{Im}{j}) \to F(C ) \to F(\operatorname{Coker}{j}).$$ Since the kernel of a map is not changed by postcomposing the map with a monomorphism (check this!), we have $$\operatorname{Ker}{(F(B) \to F(\operatorname{Im}{j}))} = \operatorname{Ker}{(F(B) \to F(\operatorname{Im}{j}) \to F(C))},$$ so by functoriality of $F$ we get a left exact sequence $0 \to F(A) \to F(B) \to F(C)$ as desired.
A natural question is: When does $\operatorname{Hom}(M,-)$ send short exact sequences to short exact sequences? In other words, when is $j_\ast = \operatorname{Hom}{(M,j)}$ an epimorphism for all short exact sequences $0\; \xrightarrow{\phantom{ij}} \; A \; \xrightarrow{i\phantom{j}}\; B\; \xrightarrow{j\phantom{i}} \; C \to 0$?
In view of left exactness of $\operatorname{Hom}{(M,-)}$ the question is: Given any morphism $h: M \to C$ and any epimorphism $j: B \to C$, when is $h$ of the form $h = j_\ast g$ for some morphism $g: M \to B$?
As you certainly know, this is precisely the definition of projective modules: $M$ is called projective if and only if $g$ always exists, for all epimorphisms $j: B \twoheadrightarrow C$ and all $h: M \to C$. For emphasis:
A module $M$ is projective if and only if $\operatorname{Hom}{(M,-)}$ is exact, that is: it sends short exact sequences to short exact sequences.
First of all, as I've said in the comments, I don't understand what you mean by "setting $M$ to be the modulo of $\operatorname{im}\alpha$", so I can't help you there.
Regarding your last paragraph, it doesn't work. In Abelian category $gf = 0\implies g = 0$ if and only if $f$ is epimorphism. For all we know, your $\alpha$ could be zero map.
Now, to the proof.
- $F$ is mono:
Let $Fs = 0$. Then $s\circ \beta = 0$ and since $\beta$ is epimorphism, $s = 0$.
- $G\circ F = 0$ (equivalently, $\operatorname{im}F\subseteq \ker G$):
$(G\circ F)(s) = s\circ \beta \circ \alpha = 0$ since $\beta\circ\alpha = 0$.
- $ \ker G\subseteq\operatorname{im}F$:
Let $Gs = 0$, i.e. $s\circ\alpha = 0$. Note that $\beta$ is cokernel of $\alpha$, so by the universal property of cokernel, there exists $s'$ such that $s'\circ\beta= s$. Thus $s\in\operatorname{im}F$.
EDIT: Let me elaborate on the last paragraph. Let's say we work with modules or Abelian groups, it works the same in any Abelian category (just without elements).
Let's start by noting that $B/\ker\beta\cong \operatorname{im}\beta = C$ by the first isomorphism theorem. Write $\varphi\colon C\to B/\ker\beta$ for the inverse of the map $b+\ker\beta\mapsto \beta(b)$. Immediately we have that the composition $\varphi\circ \beta$ is the canonical epimorphism $B \twoheadrightarrow B/\ker\beta .$ Now, since $\ker\beta=\operatorname{im}\alpha$ (it's strict equality, not just isomorphism), we can write $B/\operatorname{im}\alpha$ instead of $B/\ker\beta$.
Now, let $s\colon B\to M$ be such that $s\circ\alpha = 0$. I claim that there exists unique map $t\colon B/\operatorname{im}\alpha\to M$ to make the following diagram commute:
$\require{AMScd}$
\begin{CD}
A @>\alpha>> B @>\beta>> C @>>> 0\\
@| @| @V\varphi VV \\
A @>\alpha>> B @>\small\text{canon. epi}>> B/\operatorname{im}\alpha @>>> 0\\
@. @VsVV @VtVV \\
@. M @= M
\end{CD}
Define $t(b+\operatorname{im}\alpha) = s(b)$. I will leave the verification that this is well-defined to you. Uniqueness of such a map is obvious from the definition.
Finally, define $s'\colon C\to M$ by setting $s' = t\circ \varphi$.
All in all, what we have just proved is the following:
Let $A\stackrel{\alpha}{\to} B \stackrel{\beta}{\to} C \to 0$ be an
exact sequence. For every $s\colon B\to M$ such that $s\circ\alpha =
0$ there exists unique $s'\colon C\to M$ such that $s'\circ\beta = s$.
This is called the universal property of cokernel. In this case $\beta$ is the cokernel of $\alpha$. More precisely, the pair $(C,\beta)$ is the cokernel, but we often omit mentioning the object since it is understood from the context.
Thus, cokernel is not just an object, it is a pair of an object and a map onto it. Canonical choice for cokernel is $B \twoheadrightarrow B/\operatorname{im}\alpha $ where the arrow is the canonical epimorphism. But, in our example, we used that this canonical cokernel is isomorphic to $C$ since we really wanted a map from $C$, not from $B/\operatorname{im}\alpha$.
But, it is not enough that $B/\operatorname{im}\alpha \cong C$ to say that $C$ is cokernel, the maps onto $C$ and $B/\operatorname{im}\alpha$ must commute with the isomorphism, as you can see from the diagram. The point is, objects themselves don't matter much on their own, it's the maps that do.
Best Answer
One inclusion, namely $\ker f^*\supseteq\operatorname{im} g^*$ is obvious, because $f^*\circ g^*=(g\circ f)^*=0^*=0$. Your argument is correct.
Now let $h\in\ker f^*$, which means $f^*(h)=h\circ f=0$. This is equivalent to $\operatorname{im}f\subseteq\ker h$ and, by exactness of the original sequence, $\ker g\subseteq\ker h$.
By the homomorphism theorems, $h\colon B\to X$ induces a homomorphism $\bar{h}\colon B/\ker g\to X$ such that $h=\bar{h}\circ\pi$, where $\pi\colon B\to B/\ker g$ is the canonical map.
By assumption, $g$ induces an isomorphism $\bar{g}\colon B/\ker g\to C$ such that $g=\bar{g}\circ \pi$. Consider $k=\bar{h}\circ\bar{g}^{-1}\colon C\to X$ and compute $$ g^*(k)=k\circ g $$ Your idea is good, but can be made more precise, as you see.