"determine the normal closure" is not very clear. It seems you want to identify the isomorphism type of the normal closure.
Namely, this kernel $\mathrm{BS}_0(m,n)$ is an infinitely iterated amalgam
$$\cdots\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}\ast_\mathbf{Z}\cdots$$
where each left embedding is $k\mapsto nk$ and each right embedding is $k\mapsto mk$. This is a general fact about HNN extensions (description of the isomorphism type of the kernel of the canonical homomorphism from a given HNN-extension onto $\mathbf{Z}$), which can be found in Serre's book. In particular, this kernel $\mathrm{BS}_0(m,n)$ is not free as soon as $\max(|m|,|n|)\ge 2$. Indeed,
- if $\min(|m|,|n|)=1$ (say $m=\pm 1$ and $|n|\ge 2$), this kernel $\mathrm{BS}_0(\pm 1,n)$ is isomorphic to the infinitely generated abelian group $\mathbf{Z}[1/n]$;
- if $\min(|m|,|n|)\ge 2$, this kernel $\mathrm{BS}_0(m,n)$ contains a copy of $\mathbf{Z}^2$ (inside the amalgam $\mathbf{Z}\ast_\mathbf{Z}\mathbf{Z}=\langle a,b\mid a^m=b^n\rangle$, namely $\langle a^m,ab\rangle\simeq\mathbf{Z}^2$), hence is not free.
- (If $|m|=|n|=1$ this kernel $\mathrm{BS}_0(m,n)$ is infinite cyclic.)
On the other hand, the kernel $\mathrm{BS}_{00}(m,n)$ of the canonical homomorphism onto $\mathbf{Z}[m/n,n/m]\rtimes_{m/n}\mathbf{Z}$ is free, since it acts freely on the Bass-Serre tree.
Note: $\mathrm{BS}_0(m,n)$ is locally residually finite. Also $\mathrm{BS}_0(m,n)$ is not residually finite if $|m|,|n|$ are coprime and $\ge 2$: this is due to R. Campbell (1990 Proc AMS). I don't know if it's also non-residually-finite whenever $2\le |m|<|n|$, for instance, $(m,n)=(2,4)$.
Edit: the free subgroup $\mathrm{BS}_{00}(m,n)$ is infinitely generated as soon as $2\le |m|<|n|$. Since $2\le \min(|m|,|n|)$, it is not trivial, hence contains a loxodromic element for the action on the Bass-Serre tree. Hence it has a unique minimal nonempty subtree (the convex hull of the union of axes of loxodromic elements) for the action on the Bass-Serre tree $T$ of the HNN extension defining $\mathrm{BS}(m,n)$; hence this subtree is invariant under the whole group, and hence by vertex-transitivity, this is the whole tree.
The tree $T$ naturally carries a Busemann-function (defined up to addition with an integral constant), so that every vertex $v$ has $n+m$ adjacent vertices: $m$ vertices $w$ with $b(w)=b(v)-1$ and $n$ vertices $w$ with $b(w)=b(v)+1$. Let $G$ is the group of automorphisms of $T$ preserving $b+\mathbf{Z}$, so the locally compact group $G$ acts cocompactly on $T$ and $\mathrm{BS}(m,n)$ acts through $G$.
Now assume by contradiction that $\mathrm{BS}_{00}(m,n)$ is finitely generated: then since it acts freely and minimally on the tree $T$, the action is cocompact. Hence $\mathrm{BS}_{00}(m,n)$ is a cocompact lattice in $G$. But $G$ is not unimodular, by an easy argument (using $|m|\neq |n|$). This is a contradiction.
Best Answer
Recent results on generalized Baumslag-Solitar groups does describe what they mean (maybe in a somewhat round about way).
The type of Tits alternative is this: Finitely generated subgroups of GBS-groups are either solvable or contain non-abelian free subgroups.
The above statement follows from the theorems they mention:
Theorem 1: The second derived subgroup of a GBS-group is free.
In particular GBS-groups are solvable exactly when the second derived subgroup is trivial or $\mathbb Z$. Otherwise it contains a higher rank free subgroup, so can not be solvable.
Theorem 2: Finitely generated subgroups of GBS-group are either free or GBS-groups.
This theorem gives us a way to apply theorem 1 to finitely generated subgroups of GBS-groups. This finishes the proof of "finitely generated Tits alternative".