In response to your request in the comments for an example of a proof of an isomorphism using generators and relations, I found the following proof in Hungerford's Algebra page 67. I'm not sure if this example will satisfy what you had in mind, if not let me know I can try to dig deeper.
Let $G$ be the group defined by generators $a,b$ and relations $a^4=e$, $a^2b^{-2}=e$ and $abab^{-1}=e$. Since $Q_8$, the quaternion group of order $8$, is generated by elements $a,b$ satisfying these relations (Exercise 4.14), there is an epimorphism $\phi:G\to Q_8$ by Theorem 9.5. Hence $|G|\geq|Q_8|=8$. Let $F$ be the free group on $\{a,b\}$ and $N$ the normal subgroup generated by $\{a^4,a^2b^{-2}\}$. It is not difficult to show that every element of $F/N$ is of the form $a^ib^iN$ with $0\leq i\leq 3$ and $j=0,1$, whence $|G|=|F/N|\leq8$. Therefore $|G|=8$ and $\phi$ is an isomorphism. Thus the group defined by the given generators and relations is isomorphic to $Q_8$.
If you require uniqueness of your solution, then I don’t believe this is possible.
To shorten notation, for a set $X$ I’ll write $\sigma(X)$ to denote a word in elements of $X$. Let $X$ be a set, let $o$ be a variable, and let $\sigma(X, o)$ be any word. Then the group
$$G = \langle X, s, t~|~\sigma(X, s) = \sigma(X, t) = 1\rangle$$
fails the uniqueness condition for solutions to $\sigma(X, o)$.
So, let’s consider the case where we don’t assume uniqueness. In this case, it is (uninterestingly) possible.
The requirement that $\sigma(X, o) = 1$ for all groups means that, in particular, this must be true for the free group $F(X\cup\{o\})$. This means that $\sigma(X,o)$ must reduce to a trivial word by the definition of free groups.
The only next requirement is that $o\notin \langle X\rangle$. Thus, we can produce a situation you want in the following way: let $G$ be any group with identity element $e$, and $\sigma(X,o)$ be any word that reduces to the trivial word.
Then, the assignment $x\mapsto e$ for all $x\in X$ and $o\mapsto g$ for any $g\ne e$ in $G$ satisfies your requirements.
Best Answer
As you said, if $1\in\{|m|,|n|\}$ then you get a semidirect product, say $\mathrm{BS}(1,n)=\mathbf{Z}[1/n]\rtimes\mathbf{Z}$ for $n\neq 0$ and $=\mathbf{Z}$ for $n=0$.
Otherwise this group has a free non-abelian subgroup, and indeed its second derived subgroup is a free group (so it's residually solvable anyway).
In the cases $|m|=|n|\ge 2$ or ($mn=0$ and $\max(|m|,|n|)\ge 2$) the group is actually virtually free non-abelian. ($\mathrm{BS}(0,n)\simeq\mathbf{Z}\ast C_{|n|}$). In the remaining cases (namely $|m|\neq |n|$ and $\min(|m|,|n|)\ge 2$ it is known not to be residually finite.
A standard way to show that they're indeed non-abelian is to use that these are non-ascending HNN extensions (HNN extension by a group over an isomorphism between two proper subgroups): each such group, by Bass-Serre theory, acts on a regular tree of valency $\ge 3$ with no invariant proper subtree and no fixed point at infinity, and this forces the existence of a non-abelian free subgroup.