Now, i have this:
$\varphi:\left\{a,b\right\}\to <f,g>$ with $\varphi(a)=f$ and $\varphi(b)=g$ homomorphism.
exists unique epimorphism $\varphi F(a,b)\to <f,g>$ such that
$\varphi(w)=w$ with $w$ word in Domain, and $w$ group element in Codomain.
further, $F(a,b)/\ker\varphi\simeq <f,g>$.
Afirmation. $\ker\varphi=<< aba^{-1}b^{-2}>>$.
Obviously $<< aba^{-1}b^{-2}>>\subset \ker\varphi$.
Now, let $w\in \ker\varphi$, then $w=a^{-k}b^{m}a^{k+n}$ with
$\varphi(a^{-k}b^{m}a^{k+n})=Id$, or, equivalent, $2^nx+\frac{m}{2^k}=x$, and this implies $n=m=0$.
Therefore, $w\sim a^{-k}a^{k}\sim \epsilon\sim aba^{-1}b^{-2}\in <<aba^{-1}b^{-2}>>$.
Therefore $\ker\varphi=<<aba^{-1}b^{-2}>>$
It is correct?
"determine the normal closure" is not very clear. It seems you want to identify the isomorphism type of the normal closure.
Namely, this kernel $\mathrm{BS}_0(m,n)$ is an infinitely iterated amalgam
$$\cdots\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\cdots$$
where each left embedding is $k\mapsto nk$ and each right embedding is $k\mapsto mk$. This is a general fact about HNN extensions (description of the isomorphism type of the kernel of the canonical homomorphism from a given HNN-extension onto $\mathbf{Z}$), which can be found in Serre's book. In particular, this kernel $\mathrm{BS}_0(m,n)$ is not free as soon as $\max(|m|,|n|)\ge 2$. Indeed,
- if $\min(|m|,|n|)=1$ (say $m=\pm 1$ and $|n|\ge 2$), this kernel $\mathrm{BS}_0(\pm 1,n)$ is isomorphic to the infinitely generated abelian group $\mathbf{Z}[1/n]$;
- if $\min(|m|,|n|)\ge 2$, this kernel $\mathrm{BS}_0(m,n)$ contains a copy of $\mathbf{Z}^2$ (inside the amalgam $\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}=\langle a,b\mid a^m=b^n\rangle$, namely $\langle a^m,ab\rangle\simeq\mathbf{Z}^2$), hence is not free.
- (If $|m|=|n|=1$ this kernel $\mathrm{BS}_0(m,n)$ is infinite cyclic.)
On the other hand, the kernel $\mathrm{BS}_{00}(m,n)$ of the canonical homomorphism onto $\mathbf{Z}[m/n,n/m]\rtimes_{m/n}\mathbf{Z}$ is free, since it acts freely on the Bass-Serre tree.
Note: $\mathrm{BS}_0(m,n)$ is locally residually finite. Also $\mathrm{BS}_0(m,n)$ is not residually finite if $|m|,|n|$ are coprime and $\ge 2$: this is due to R. Campbell (1990 Proc AMS). I don't know if it's also non-residually-finite whenever $2\le |m|<|n|$, for instance, $(m,n)=(2,4)$.
Edit: the free subgroup $\mathrm{BS}_{00}(m,n)$ is infinitely generated as soon as $2\le |m|<|n|$. Since $2\le \min(|m|,|n|)$, it is not trivial, hence contains a loxodromic element for the action on the Bass-Serre tree. Hence it has a unique minimal nonempty subtree (the convex hull of the union of axes of loxodromic elements) for the action on the Bass-Serre tree $T$ of the HNN extension defining $\mathrm{BS}(m,n)$; hence this subtree is invariant under the whole group, and hence by vertex-transitivity, this is the whole tree.
The tree $T$ naturally carries a Busemann-function (defined up to addition with an integral constant), so that every vertex $v$ has $n+m$ adjacent vertices: $m$ vertices $w$ with $b(w)=b(v)-1$ and $n$ vertices $w$ with $b(w)=b(v)+1$. Let $G$ is the group of automorphisms of $T$ preserving $b+\mathbf{Z}$, so the locally compact group $G$ acts cocompactly on $T$ and $\mathrm{BS}(m,n)$ acts through $G$.
Now assume by contradiction that $\mathrm{BS}_{00}(m,n)$ is finitely generated: then since it acts freely and minimally on the tree $T$, the action is cocompact. Hence $\mathrm{BS}_{00}(m,n)$ is a cocompact lattice in $G$. But $G$ is not unimodular, by an easy argument (using $|m|\neq |n|$). This is a contradiction.
Best Answer
I'll expand my comment into an answer. We can assume that $1 \le p\le |q|$.
If $p \ge 2$, then the free reduction of a word in the generators $a$ and $c := bab^{-1}$ does not involve any power $b^k$ of $b$ with $|k| \ge 2$, so by Britton's lemma on HNN-extensions, the subgroup $\langle a,c \rangle$ is freely generated by $a$ and $c$, and is not virtually abelian.
When $p=1$ and $|q| \ge 2$, the normal closure of $\langle b \rangle$ in $G$ is abelian but not finitely generated. Since any subgroup of a finitely generated virtually abelian group is itself finitely generated, $G$ cannot be virtually abelian in this case.
When $p=|q|=1$ it is easy to see that $G$ is either free abelian of rank $2$ or it has a subgroup of index $2$ (i.e.$\langle a^2,b \rangle$) that is free abelian, so $G$ is virtually abelian in the case.
When $|p|=|q|$ we can describe the structure of the group very precisely, but it is still not virtually abelian when $|p|>1$. For example, if $p=q>1$, then the normal closure of $\langle a \rangle$ in $G$ is a free group of rank $q-1$, with a free basis cycled by conjugation by $b$.