For some $g \geq 2$, let $\Gamma_g$ be the fundamental group of a closed genus $g$ surface and let $S_g=\{a_1,b_1,\ldots,a_g,b_g\}$ be the usual generating set for $\Gamma_g$ satisfying the surface relation $[a_1,b_1]\cdots[a_g,b_g]=1$. Dehn proved the following famous theorem.
Theorem : Let $w$ be a reduced word in the generators $S_g$ such that $w$ represents the identity in $\Gamma_g$. Then $w$ contains a subword $r_1$ of length $k \geq 2g+1$ such that there exists a word $r_2$ of length $4g-k$ with $r_1 r_2$ a cyclic permutation of $[a_1,b_1]\cdots[a_g,b_g]$.
This leads to Dehn's algorithm for solving the word problem in a surface group. Namely, start with a word $w$ which is reduced. If $w$ does not contain a subword $r_1$ as in the Theorem, then $w$ does not represent the identity in $\Gamma_g$. Otherwise, we can replace the subword $r_1$ of $w$ with $r_2$, shortening $w$. After possibly reducing $w$, we do the above again. We stop if we have reduced $w$ to the empty word.
There are many sources for Dehn's algorithm; for instance, Stillwell's book on geometric topology and combinatorial group theory gives a proof using elementary topology, and Lyndon-Schupp give a proof based on small cancellation theory. However, many sources say that Dehn's original proof used hyperbolic geometry. I've heard people tell me that he studied the usual tessalation of the hyperbolic plane by regular $4g$-gons and applied something like Gauss-Bonnet; however, I've never managed to reconstruct this kind of proof. Can someone either spell it out or give a (modern) reference?
Best Answer
Dehn gave at least three solutions of the conjugacy problem for surface groups, which can be found in my translation in the book Papers on Group Theory and Topology (Springer 1986), Papers 2, 4, and 5.
The first is based on an idea of Poincaré: lifting a curve to the universal cover, which is the disk model of the hyperbolic plane, replacing it by the hyperbolic straight line with the same endpoints, then projecting this line back to the surface as a "geodesic representative" of the original curve. Two curves are free homotopic (and the corresponding group elements are conjugate) if and only if they have the same geodesic representative.
This unpublished proof is conceptually simple, but it is not clear how to determine whether two curves have the same geodesic representative. Dehn's first published proof (Paper 4, 1912) worked out a way to do this, arriving in a roundabout way at Dehn's algorithm.
Shortly afterward (Paper 5, still in 1912) Dehn noticed that the algorithm follows easily from combinatorial properties of the tessellation of the universal cover, and the hyperbolic metric is irrelevant. This is essentially the proof we use today.