I want to determine all fractional transformation $F:\mathbb{C}\to\mathbb{C}$ which maps $\mathbb{R}$ on $\mathbb{R}$ and the unit circle on the unit circle. Let $\begin{pmatrix}a&b\\c&d
\end{pmatrix}$ be such a linear transformation.
For $x\in\mathbb{R}$, we must have that $\frac{ax+b}{cx+d}\in\mathbb{R}.$ Then $$\frac{ax+b}{cx+d}=
\frac{\overline{ax+b}}{\overline{cx+d}}=\frac{\overline{a}x+\overline{b}}{\overline{c}x+\overline{d}}.
$$
Can we conclue from this that $\begin{pmatrix}a&b\\c&d\end{pmatrix}=\lambda\begin{pmatrix}\overline{a}&\overline{b}\\\overline{c}&\overline{d}\end{pmatrix}$ for some $\lambda\in\mathbb{C}$ met $|\lambda|=1$. Then also for $y\in S^1$, we have that $$\bigg|\frac{ay+b}{cy+d}\bigg|=1$$
how can I continue?
Best Answer
As you have noted in your comment, $F$ either fixes $1$ and $-1$ or swaps them. Replacing $F$ by $-F$ if necessary, we can assume $F$ fixes $1$ and $-1$. Then we have $$ \frac{a\cdot 1 + b}{c \cdot 1 + d} = 1 \quad\quad \frac{a\cdot -1 + b}{c \cdot -1 + d} = -1 $$ So: $ a + b = c + d $ and $ b - a = c - d $ Adding these equations gives $2b = 2c$ and subtracting them gives $2a = 2d$. If $b = 0$, $F(z) = z$, otherwise we can divide through by $b$ and so assume $b = 1$. Thus the matrix of $F$ is either $$ \begin{pmatrix}1&0\\ 0 & 1\end{pmatrix} \quad\mbox{or} \quad \begin{pmatrix}a&1\\ 1 & a\end{pmatrix} $$ for some $a$. However, $F$ maps the unit circle to itself, and as $F$ maps circles to circles (or lines), this will hold iff $|F(i)| = 1$, i.e., iff $ |ai + 1| = |i + a| $ which holds iff $ (ai+1)(\overline{ai+1}) = (i + a)(\overline{i + a}) $ but $$ \begin{align*} (ai+1)(\overline{ai+1}) &= a\overline{a} + (a - \overline{a})i + 1 \\ (i + a)(\overline{i + a}) &= a\overline{a} + (\overline{a} - a)i + 1 \end{align*} $$ So $|F(i)| = 1$ iff $a - \overline{a} = 0$, i.e., iff $a \in \Bbb{R}$.
We conclude that the linear transformation $F$ maps $\Bbb{R}$ to $\Bbb{R}$ and $S^1$ to $S^1$ iff its matrix is either $$ \pm \begin{pmatrix}1&0\\ 0 & 1\end{pmatrix} \quad\mbox{or} \quad \pm \begin{pmatrix}a&1\\ 1 & a\end{pmatrix} $$ for some $a \in \Bbb{R}$. Note that this is degenerate when $a = 1$. For $a \neq 1$ it maps $\Bbb{R}$ onto $\Bbb{R}$ and $S^1$ onto $S^1$