In Groups and Geometry by Lyndon, Chapter 9 part 3 (page 165) started by introducing a new group (denoted as $\widetilde{H}$ below) and a theorem.
Let $H$ denote the hyperbolic group and $H^{+} = \{ x + iy : y > 0 \}$. It was shown in a previous part that $H \cong LF(2,\mathbb{R})$. Now let $\widetilde{H}$ be the group containing reflections in all hyperbolic lines of $H^{+}$. Here, $H \leq \widetilde{H}$.
Edit: $\widetilde{H}$ is defined to be the stabilizer of $\mathbb{R} \cup \{\infty\}$ in $M$, where $M$ is the inversive group. The inversive group $M$ is defined to be the group of all transformations of $\mathbb{C} \cup \{\infty\}$ generated by the inversions in circles and Euclidean lines.
Theorem. The group $\widetilde{H}$ is generated by all inversions in hyperbolic lines of $H^{+}$, and $H$ is the subgroup of $\widetilde{H}$ generated by all products of two such inversions. In particular, if $\alpha : z \mapsto -\overline{z}$ is the inversion in the (upper half of) the imaginary axis, then $\widetilde{H} = \langle \alpha \rangle \cdot H$, the semidirect product of $H$ by the group $\langle \alpha \rangle$ of order 2.
My first question is how exactly do the elements of $\widetilde{H}$ look like? Or how can you define it using set notations? We have discussed inversions in circles and linear fractionals but not reflections in hyperbolic lines. Previous parts of the chapter did not discuss it as well. It is not clear to me how the inversions generate $\widetilde{H}$.
Second, I do not see why $H$ is generated by all products of two such inversions. Should this be clear if I know explicitly the elements of $\widetilde{H}$?
Best Answer
Let me first say that the notation of this book is a little nonstandard. If I understand correctly, $H = LF(2,\mathbb R)$ is the group usually denoted $PSL(2,\mathbb R)$, and I will proceed with that assumption. (I will guess that "LF" means "linear fractional", which makes sense but unfortunately is not a standard notation nowadays).
You can understand reflections in hyperbolic lines using your understanding of inversions in circles. A hyperbolic line $L \subset H^+$ is simply a semicircle whose endpoints lie on the real line $\{x+0i\}$. Reflection of $H^+$ across a hyperbolic line $L$ is defined to be the restriction to $H^+$ of inversion across the unique circle $C$ that contains $L$.
Regarding your question about the elements of $\widetilde H$, perhaps you have learned the classification of elements of the group of isometries of the Euclidean plane: every isometry of the Euclidean plane is either the identity, or a rotation of some angle centered at some point, or a translation of some distance along some line, or a reflection across some line, or a glide reflector of some distance along some line. Classification of elements of $\widetilde H$ proceeds in the exact same language: every element in $H$ itself is either the identity, or a rotation of some angle centered at some point, or a translation of some distance along some line; each additional elements of $\widetilde H$ that is not in $H$ is either the reflections across some line, or a glide reflector of some distance along some line.
Regarding your question regarding why $\widetilde H$ is generated by all reflections in hyperbolic lines, unfortunately the question does not quite make sense because you have not actually given a mathematical definition of $\widetilde H$. It is not well-defined to say that $\widetilde H$ is the group containing reflections in all hyperbolic lines of $H^+$; I can think of many groups that contain those reflections, for example the group of all permutations of the point set $H^+$. Perhaps you have left something out of the definition of $\widetilde H$, and probably it would be worthwhile to examine the definition given in Lyndon's book more closely.