Now, i have this:
$\varphi:\left\{a,b\right\}\to <f,g>$ with $\varphi(a)=f$ and $\varphi(b)=g$ homomorphism.
exists unique epimorphism $\varphi F(a,b)\to <f,g>$ such that
$\varphi(w)=w$ with $w$ word in Domain, and $w$ group element in Codomain.
further, $F(a,b)/\ker\varphi\simeq <f,g>$.
Afirmation. $\ker\varphi=<< aba^{-1}b^{-2}>>$.
Obviously $<< aba^{-1}b^{-2}>>\subset \ker\varphi$.
Now, let $w\in \ker\varphi$, then $w=a^{-k}b^{m}a^{k+n}$ with
$\varphi(a^{-k}b^{m}a^{k+n})=Id$, or, equivalent, $2^nx+\frac{m}{2^k}=x$, and this implies $n=m=0$.
Therefore, $w\sim a^{-k}a^{k}\sim \epsilon\sim aba^{-1}b^{-2}\in <<aba^{-1}b^{-2}>>$.
Therefore $\ker\varphi=<<aba^{-1}b^{-2}>>$
It is correct?
"determine the normal closure" is not very clear. It seems you want to identify the isomorphism type of the normal closure.
Namely, this kernel $\mathrm{BS}_0(m,n)$ is an infinitely iterated amalgam
$$\cdots\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\cdots$$
where each left embedding is $k\mapsto nk$ and each right embedding is $k\mapsto mk$. This is a general fact about HNN extensions (description of the isomorphism type of the kernel of the canonical homomorphism from a given HNN-extension onto $\mathbf{Z}$), which can be found in Serre's book. In particular, this kernel $\mathrm{BS}_0(m,n)$ is not free as soon as $\max(|m|,|n|)\ge 2$. Indeed,
- if $\min(|m|,|n|)=1$ (say $m=\pm 1$ and $|n|\ge 2$), this kernel $\mathrm{BS}_0(\pm 1,n)$ is isomorphic to the infinitely generated abelian group $\mathbf{Z}[1/n]$;
- if $\min(|m|,|n|)\ge 2$, this kernel $\mathrm{BS}_0(m,n)$ contains a copy of $\mathbf{Z}^2$ (inside the amalgam $\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}=\langle a,b\mid a^m=b^n\rangle$, namely $\langle a^m,ab\rangle\simeq\mathbf{Z}^2$), hence is not free.
- (If $|m|=|n|=1$ this kernel $\mathrm{BS}_0(m,n)$ is infinite cyclic.)
On the other hand, the kernel $\mathrm{BS}_{00}(m,n)$ of the canonical homomorphism onto $\mathbf{Z}[m/n,n/m]\rtimes_{m/n}\mathbf{Z}$ is free, since it acts freely on the Bass-Serre tree.
Note: $\mathrm{BS}_0(m,n)$ is locally residually finite. Also $\mathrm{BS}_0(m,n)$ is not residually finite if $|m|,|n|$ are coprime and $\ge 2$: this is due to R. Campbell (1990 Proc AMS). I don't know if it's also non-residually-finite whenever $2\le |m|<|n|$, for instance, $(m,n)=(2,4)$.
Edit: the free subgroup $\mathrm{BS}_{00}(m,n)$ is infinitely generated as soon as $2\le |m|<|n|$. Since $2\le \min(|m|,|n|)$, it is not trivial, hence contains a loxodromic element for the action on the Bass-Serre tree. Hence it has a unique minimal nonempty subtree (the convex hull of the union of axes of loxodromic elements) for the action on the Bass-Serre tree $T$ of the HNN extension defining $\mathrm{BS}(m,n)$; hence this subtree is invariant under the whole group, and hence by vertex-transitivity, this is the whole tree.
The tree $T$ naturally carries a Busemann-function (defined up to addition with an integral constant), so that every vertex $v$ has $n+m$ adjacent vertices: $m$ vertices $w$ with $b(w)=b(v)-1$ and $n$ vertices $w$ with $b(w)=b(v)+1$. Let $G$ is the group of automorphisms of $T$ preserving $b+\mathbf{Z}$, so the locally compact group $G$ acts cocompactly on $T$ and $\mathrm{BS}(m,n)$ acts through $G$.
Now assume by contradiction that $\mathrm{BS}_{00}(m,n)$ is finitely generated: then since it acts freely and minimally on the tree $T$, the action is cocompact. Hence $\mathrm{BS}_{00}(m,n)$ is a cocompact lattice in $G$. But $G$ is not unimodular, by an easy argument (using $|m|\neq |n|$). This is a contradiction.
Best Answer
The only nilpotent Baumslag-Solitar group is $\mathbb{Z}^2=\langle a,b\mid [a,b]\rangle$. More generally:
Theorem. A one-relator group is nilpotent if and only if it is abelian, if and only if it is a cyclic group or $\mathbb{Z}^2=\langle a,b\mid [a,b]\rangle$.
Proof. Let $G$ be a one-relator group which is not cyclic nor isomorphic to $\mathbb{Z}^2$, and which has non-trivial centre. Then $G$ is two generated and its centre is infinite cyclic [1]. Moreover, Pietrowski [2] proved that such a group $G$ admits a presentation $$\mathcal{P}:=\langle x_1, \ldots, x_n\mid x_1^{p_1}=x_2^{q_1}, \ldots, x_{n-1}^{p_{n-1}}=x_n^{q_{n-1}}\rangle.$$ Indeed, in this form the centre is the subgroup $\cap\langle x_i\rangle$. (Given a presentation of this form it is not easy to verify if it defines a one-relator group, although an algorithm exists [3], but a simple example is $\langle a,b; a^m=b^n\rangle$.)
Now, abelianise this presentation $\mathcal{P}$. Every one-relator group surjects onto $\mathbb{Z}$, and so this abelianisation also maps onto $\mathbb{Z}$. Therefore, some generator $x_i$ has infinite order in the abelianisation. As the centre is wholly contained in $\langle x_i\rangle$, the centre intersects the derived subgroup trivially. This means that $G$ is not nilpotent, as required, because non-abelian nilpotent groups have a centre which intersects the derived subgroup non-trivially (see, for example, here). QED
In fact, we have a stronger result:
Corollary. A one-relator presentation $\langle \mathbf{x}\mid R\rangle$ defines a nilpotent group if and only if one of the following holds
I won't detail a proof, but the point is that in each case the group is "obviously" nilpotent (1. and 3. correspond to cyclic groups, 2. is $\mathbb{Z}^2$), while certain well-known results in combinatorial group theory give us that a one-relator presentation of a cyclic group or $\mathbb{Z}^2$ must have one of these forms (for example, the main result for (2) is Theorem 3.9 of the book Combinatorial Group Theory by Magnus, Karrass and Solitar).
[1] Murasugi, K. "The center of a group with a single defining relation." Mathematische Annalen 155.3 (1964): 246-251.
[2] Pietrowski, A. "The isomorphism problem for one-relator groups with non-trivial centre." Mathematische Zeitschrift 136.2 (1974): 95-106.
[3] Metaftsis, V. "An algorithm for stem products and one-relator groups." Proceedings of the Edinburgh Mathematical Society 42.1 (1999): 37-42.