Now, i have this:
$\varphi:\left\{a,b\right\}\to <f,g>$ with $\varphi(a)=f$ and $\varphi(b)=g$ homomorphism.
exists unique epimorphism $\varphi F(a,b)\to <f,g>$ such that
$\varphi(w)=w$ with $w$ word in Domain, and $w$ group element in Codomain.
further, $F(a,b)/\ker\varphi\simeq <f,g>$.
Afirmation. $\ker\varphi=<< aba^{-1}b^{-2}>>$.
Obviously $<< aba^{-1}b^{-2}>>\subset \ker\varphi$.
Now, let $w\in \ker\varphi$, then $w=a^{-k}b^{m}a^{k+n}$ with
$\varphi(a^{-k}b^{m}a^{k+n})=Id$, or, equivalent, $2^nx+\frac{m}{2^k}=x$, and this implies $n=m=0$.
Therefore, $w\sim a^{-k}a^{k}\sim \epsilon\sim aba^{-1}b^{-2}\in <<aba^{-1}b^{-2}>>$.
Therefore $\ker\varphi=<<aba^{-1}b^{-2}>>$
It is correct?
"determine the normal closure" is not very clear. It seems you want to identify the isomorphism type of the normal closure.
Namely, this kernel $\mathrm{BS}_0(m,n)$ is an infinitely iterated amalgam
$$\cdots\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\cdots$$
where each left embedding is $k\mapsto nk$ and each right embedding is $k\mapsto mk$. This is a general fact about HNN extensions (description of the isomorphism type of the kernel of the canonical homomorphism from a given HNN-extension onto $\mathbf{Z}$), which can be found in Serre's book. In particular, this kernel $\mathrm{BS}_0(m,n)$ is not free as soon as $\max(|m|,|n|)\ge 2$. Indeed,
- if $\min(|m|,|n|)=1$ (say $m=\pm 1$ and $|n|\ge 2$), this kernel $\mathrm{BS}_0(\pm 1,n)$ is isomorphic to the infinitely generated abelian group $\mathbf{Z}[1/n]$;
- if $\min(|m|,|n|)\ge 2$, this kernel $\mathrm{BS}_0(m,n)$ contains a copy of $\mathbf{Z}^2$ (inside the amalgam $\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}=\langle a,b\mid a^m=b^n\rangle$, namely $\langle a^m,ab\rangle\simeq\mathbf{Z}^2$), hence is not free.
- (If $|m|=|n|=1$ this kernel $\mathrm{BS}_0(m,n)$ is infinite cyclic.)
On the other hand, the kernel $\mathrm{BS}_{00}(m,n)$ of the canonical homomorphism onto $\mathbf{Z}[m/n,n/m]\rtimes_{m/n}\mathbf{Z}$ is free, since it acts freely on the Bass-Serre tree.
Note: $\mathrm{BS}_0(m,n)$ is locally residually finite. Also $\mathrm{BS}_0(m,n)$ is not residually finite if $|m|,|n|$ are coprime and $\ge 2$: this is due to R. Campbell (1990 Proc AMS). I don't know if it's also non-residually-finite whenever $2\le |m|<|n|$, for instance, $(m,n)=(2,4)$.
Edit: the free subgroup $\mathrm{BS}_{00}(m,n)$ is infinitely generated as soon as $2\le |m|<|n|$. Since $2\le \min(|m|,|n|)$, it is not trivial, hence contains a loxodromic element for the action on the Bass-Serre tree. Hence it has a unique minimal nonempty subtree (the convex hull of the union of axes of loxodromic elements) for the action on the Bass-Serre tree $T$ of the HNN extension defining $\mathrm{BS}(m,n)$; hence this subtree is invariant under the whole group, and hence by vertex-transitivity, this is the whole tree.
The tree $T$ naturally carries a Busemann-function (defined up to addition with an integral constant), so that every vertex $v$ has $n+m$ adjacent vertices: $m$ vertices $w$ with $b(w)=b(v)-1$ and $n$ vertices $w$ with $b(w)=b(v)+1$. Let $G$ is the group of automorphisms of $T$ preserving $b+\mathbf{Z}$, so the locally compact group $G$ acts cocompactly on $T$ and $\mathrm{BS}(m,n)$ acts through $G$.
Now assume by contradiction that $\mathrm{BS}_{00}(m,n)$ is finitely generated: then since it acts freely and minimally on the tree $T$, the action is cocompact. Hence $\mathrm{BS}_{00}(m,n)$ is a cocompact lattice in $G$. But $G$ is not unimodular, by an easy argument (using $|m|\neq |n|$). This is a contradiction.
Best Answer
The answer is no. Let us show something stronger: for every $u\neq 1$, the set of $n$ such that $u$ has an $n$-root is bounded. Let me show this for $$G=\mathrm{BS}(m,n)=\langle t,x\mid tx^mt^{-1}=x^n\rangle$$ whenever neither $m/n$ nor $m/n$ is an integer.
Indeed, $G$ acts on its Bass-Serre tree $T$, with a vertex $v_0$ with stabilizer $\langle x\rangle$. So every vertex $v$ has cyclic stabilizer, with a unique generator $\xi_v$ conjugate to $x$.
Let $u$ satisfy the previous property. This property forces $u$ to act as an elliptic element (indeed if $u$ were loxodromic of translation length $R$, it could not have $p$-roots for $p>R$).
The Bass-Serre tree has a unique height function $h$ satisfying: $h(tv)=h(v)+1$ and $h(xv)=h(v)$ for all vertices $v$. Then every vertex $v$ has exactly $n+m$ neighbors, consisting of two $G_v$-orbits: one orbit of $n$ neighbors $v'$ with $h(v')=h(v)+1$ (called right neighbors) and one orbit of $m$ neighbors $v'$ with $h(v')=h(v)-1$ (called left neighbors). If $v$ is a vertex and $v'$ is a right neighbor, then we have the formula $\xi_v^n=\xi_{v'}^m$.
Let $s\in G$ be an elliptic element (for the action on $T$). Let $T^s$ be the tree of fixed points of $s$. For every vertex $v\in T^s$, we have $s\in G_v$, so $s=\xi_v^{q_u(v)}$ for some unique $q_s(v)\in\mathbf{Z}$. If $v,v'\in T^s$ with $v'$ right neighbor of $v$, then by the previous formula, we have $q_s(v')=\frac{m}{n}q_s(v)$. Hence by connectedness of $T^s$, for any two vertices $v,v'\in T^s$ we have $q_s(v')=\left(\frac{m}{n}\right)^{h(v')-h(v)}q_s(v)$.
If neither $m/n$ nor $n/m$ is an integer and $s\neq 1$, it follows that $h(T^s)$ is bounded, and in turn, it follows that $q_s$ is bounded on $T^s$.
Now let $u\neq 1$ have roots of unbounded order. As we have seen, $u$ is elliptic. Then for some $R$ we have $|q_u(v)|\le R$ for all $v\in T^u$. Suppose that $u=s^N$. So $s$ is elliptic as well, so fixes a vertex $v$, and necessarily $v\in T^u$. It follows that $N\le R$.
Comment: if $n/m=\pm 1$, the same conclusion holds by a distinct (easier) argument, while if exactly one of $m/n$ or $n/m$ is an integer $r$ (so $|r|\ge 2$), the conclusion fails as one can then check that $\mathrm{BS}(m,n)$ then contains an isomorphic copy of $\mathbf{Z}[1/r]$: in $\mathrm{BS}(m,mr)$ we have $x^m=(t^{-p}x^mt^p)^{r^p}$ for all positive integers $p$.
Another fact (to address Derek's comments): let $I$ be an infinite set of positive integers. Write $G^n=\{g^n:g\in G\}$. Say that a group $G$ satisfies Property $P_I$ if $\bigcap_{n\in I} G^n=\{1\}$, and $g^n=1$ implies $g=1$ for every $n\in I$. Then Property $P_I$ passes to group extensions.
If $G$ is free, then it satisfies $P_I$ for every infinite $I$.
If $G=\mathbf{Z}[1/k]$, then it satisfies $P_I$ for every infinite $I$ of integers coprime to $k$.
Define $k=mn/\gcd(m,n)^2$. Then there is an obvious homomorphism $\mathrm{BS}(m,n)\to\mathbf{Z}[1/k]\rtimes_{n/m}\mathbf{Z}$. It follows that for every infinite $I$ of elements coprime to $k$, $\mathrm{BS}(m,n)$ has Property $P_I$. In particular, every nontrivial element is divisible by only finitely many primes.