Define $$\phi_a(z) = \frac{z-a}{1-\overline{a}z}, \qquad \rho_\alpha(z) = e^{i\alpha}z,$$ with $|a|<1$ and $\alpha \in \mathbb{R}$, so that $\phi_a \circ \rho_\alpha$ is a holomorphic automorphism of the unit disc.
There are then three questions:
(I) Find $c$ and $\gamma$ such that
$(\phi_a \circ \rho_\alpha)\circ(\phi_b \circ \rho_\beta) = \phi_c \circ \rho_\gamma$.
(II) Compute $(\phi_a \circ \rho_\alpha – \rho_\alpha \circ \phi_a)(z) $.
(III) Using knowledge of the automorphisms of the unit disc, describe the automorphisms of the upper half plane $\mathbf{H}$ as fractional linear transformations.
I mistakenly thought that part (I) would come down to just some simple algebraic manipulation, but I've gone through about a dozen pages of attempts and I just can't seem to get it in the form required to extract a suitable $c$ and $\gamma$. I'm really hoping that there's some neat trick that will make the answer just fall out.
For part (II), I've gone through it and obtained a fairly horrendous answer, but I was wondering whether there is actually a tidy answer, namely one tidier than:
$$(\phi_a \circ \rho_\alpha – \rho_\alpha \circ \phi_a)(z)=\frac{\overline{a} e^{i\alpha} z \left( e^{i 2 \alpha} z – z – a e^{i 2 \alpha}\right) + a \overline{a} z + a e^{i \alpha} – a}{1 – \overline{a}z – \overline{a}e^{i \alpha}z + (\overline{a})^2 e^{i \alpha} z^2}.$$
Finally, for (III), I looked it up and the automorphisms of the upper half plane are $$z \mapsto \frac{az+b}{cz+d}, \quad ad-bc=1.$$
However, my interpretation of the question is to map the upper half plane to the unit disc, apply an automorphism and then map it back, i.e. for any $f \in \text{Aut}(\mathbf{H})$, $f = (\xi^{-1} \circ g \circ \xi) (z)$. So what I did was define $\xi(z) = \frac{z-i}{z+1}$, $g(z) = e^{i \alpha} \frac{z-a}{1-\overline{a}z}$, and then of course $\xi^{-1}(z) = \frac{z+1}{i z – i}$. Unfortunately for me, though, when I perform this composition I can't get it in a form commensurate with what I already know to be the answer.
Best Answer
Concerning (I) and (II), I second the advice given by John Adamski: it is easier to multiply matrices of the form $\begin{pmatrix} a & b \\c & d \end{pmatrix}$ then to compose maps of the form $\dfrac{az+b}{cz+d}$. The result is the same, see here.
Concerning (II), I am surprised that someone wanted to know $\phi_a \circ \rho_\alpha - \rho_\alpha \circ \phi_a$. Subtracting two automorphisms of the unit disk does not look like a natural operation to me. Maybe they meant the multiplicative commutator, $\phi_a \circ \rho_\alpha \circ (\rho_\alpha \circ \phi_a)^{-1}$?
A typo here: $\xi(z)=\dfrac{z−i}{z+i}$ would be correct. Not sure if this is where the problem was. Anyway, as in (III) it is better to work with matrices and compute $$\begin{pmatrix} 1 & -i \\1 & i \end{pmatrix}^{-1} \begin{pmatrix} a & b \\c & d \end{pmatrix}\begin{pmatrix} 1 & -i \\1 & i \end{pmatrix} $$