Let $M$ be a finite dimensional Riemannian manifold and $\operatorname{Iso}(M)$ be its set of isometries. It can be shown that $\operatorname{Iso}(M)$ is a finite dimensional manifold with a metric as defined below:
Consider $(n + 1)$ points on $M$ so close together that $n$ of them lie in an normal neighborhood of the other, and if the points are linearly independent (i.e. not in the same $(n-1)$-dimensional geodesic hypersurface). Then the distance $d(f, \tilde f)$ between two isometries $f$ and $\tilde f$ will be defined as the maximum of the distance $d_i[f(x), \tilde f(x)]$ as $x$ ranges over the given set of $n+1$ points. This distance can be shown to satisfy the usual metric axioms. Here $d_i$ is of course the induced metric on $M$ (Riemannian distance fucntion)
Given $\operatorname{Iso}(M)$ is now a metric space with metric $d$ as defined, we thus get a natural metric topology for $\operatorname{Iso}(M)$. That is open sets are all subsets that can be realized as the unions of open balls of form $B(f_0, r) = \{f \in \operatorname{Iso}(M): d(f_0,f)< r\}$ where $f_0 \in \operatorname{Iso}(M)$ and $r>0$.
$\ $
I am trying to prove that $$\mathscr M: \operatorname{Iso}(M) \times \operatorname{Iso}(M) \rightarrow \operatorname{Iso}(M), \, (f,g) \mapsto f \circ g$$ is continuous in the metric topology of $\operatorname{Iso}(M)$.
Attempt: Munkres Topology Section 46 Page 287
Let $Y$ be locally compact Hausdorff, and $X$ and $Z$ general spaces. Also let $\mathscr{C}(X,Y),\,\mathscr{C}(Y,Z),$ and $\mathscr{C}(X,Z)$ denote the spaces of continuous functions from the respective spaces with the compact open topology. Then the composition map
$$\mathscr M: \mathscr{C}(X,Y) \times\mathscr{C}(Y,Z)\rightarrow\mathscr{C}(X,Z)$$
is continuous.
The above is a proven statement, and can be assumed for now. In the statement, $X$ and $Z$ can be replaced with $M$ which has metric topology (and hence manifold topology) and thus is a general space. Further, $Y$ can also be replaced with $M$ as it is locally compact Hausdorff as a manifold. So we end up with $\mathscr{C}(M,M)$ for all 3. Furthermore, as isometries are continuous, we get that $\operatorname{Iso}(M) \subset \mathscr{C}(M,M)$. Thus we end up with the following:
$\mathscr M: \operatorname{Iso}(M) \times \operatorname{Iso}(M) \rightarrow \operatorname{Iso}(M)$ is continuous
Idea: The compact-open topology and metric topology are the same in case of $\operatorname{Iso}(M)$ under these conditions because the topologies of every space involved here comes from same $d_i$ (Riemannian distance function as defined previously)
$\ $
Q) So I'm looking for a proof that the CO Topology and metric topology are the same for $\operatorname{Iso}(M)$.
Alternatively (and preferably)
Q) Is there a direct way to show continuity of $\mathscr M$ in the metric topology of $\operatorname{Iso}(M)$ (i.e. showing inverse of an open set in metric topology of $\operatorname{Iso}(M)$ is always open in $\operatorname{Iso}(M)\times\operatorname{Iso}(M)$, or any of the equivalent definitions of metric continuity) ?
$\ $
Best Answer
In this question you already showed that $\operatorname{Iso}(M)$ is a group and the first part of the accepted answer given by Abcde refers to another question to show continuity of the inversion $$\iota: \operatorname{Iso}(M) \to \operatorname{Iso}(M), \, f \mapsto f^{-1},$$ so I will use these two facts in the following answer. Now let $S$ denote the set of the $n+1$ points defining the metric on $\operatorname{Iso}(M)$. For $g, f, \tilde{f} \in \operatorname{Iso}(M)$ we have $$d(g \circ f, g \circ \tilde{f}) = \max_{x \in S} d_i(g(f(x)), g(\tilde{f}(x))) = \max_{x \in S} d_i(f(x), \tilde{f}(x)) = d(f, \tilde{f})$$ since $g$ is an isometry, so the map $$\mathscr{M}(g, -): \operatorname{Iso}(M) \to \operatorname{Iso}(M), \, f \mapsto g \circ f$$ is a $d$-isometry and in particular continuous for any $g \in \operatorname{Iso}(M).$ Furthermore $$f \circ g = \left((f \circ g)^{-1}\right)^{-1} = (g^{-1} \circ f^{-1})^{-1}, \text{ i.e. } \mathscr{M}(f,g) = \iota(\mathscr{M}(g^{-1}, \iota(f)),$$ so the continuity of the inversion and the above shows that $$\mathscr{M}(-,g): \operatorname{Iso}(M) \to \operatorname{Iso}(M), \, f \mapsto f \circ g$$ is continuous for $g \in \operatorname{Iso}(M)$ since $\mathscr{M}(-,g) = \iota \circ \mathscr{M}(g^{-1}, -) \circ \iota$.
Now let $(f_n)_{n \in \mathbb{N}}, (g_n)_{n \in \mathbb{N}}$ be sequences in $\operatorname{Iso}(M)$ and $f,g \in \operatorname{Iso}(M)$ such that $$\lim_{n \to \infty} d(f_n, f) = \lim_{n \to \infty} d(g_n, g) = 0.$$ Then the continuity of $\mathscr{M}(-,g)$ implies $\lim_{n \to \infty} d(f_n \circ g, f \circ g) = 0$. Furthermore we can use the triangle inequality of $d$ and the fact that $\mathscr{M}(f_n, -)$ is a $d$-isometry for any $n \in \mathbb{N}$ to obtain $$d(f_n \circ g_n, f \circ g) \leq d(f_n \circ g_n, f_n \circ g) + d(f_n \circ g, f \circ g) = d(g_n, g) + d(f_n \circ g, f \circ g),$$ so $\lim_{n \to \infty} d(f_n \circ g_n, f \circ g) = 0$. Hence $$\mathscr{M}: \operatorname{Iso}(M) \times \operatorname{Iso}(M) \to \operatorname{Iso}(M), \, (f,g) \mapsto f \circ g$$ is sequentially continuous and therefore continuous on the metric space $\operatorname{Iso}(M) \times \operatorname{Iso}(M)$.