Ok, I think I worked this out. The key insight, oddly enough, was that $\operatorname{Diff}(M)$ is first countable. That helps immensely, because now instead of using the "preimages of open sets are open" definition of continuity, I can use convergent sequences. This simplified things enough for me to understand what was going on.
[To see it is first countable, it's pretty easy to cover $M$ with a countable basis of precompact charts $\{U_i\}$, and then to check that the open sets
$$ N_{1/m}^r(f; \overline{U_i},U_j,U_k) $$
Form a countable subbasis around $f$, where all the subscripts and superscript are positive integers.]
Now if $f_n\xrightarrow{\operatorname{Diff}}f$, then in terms of the topology described in the question, this means that for any tuple $(\epsilon, r, K, U, V)$, there is a corresponding $N$ such that for $n\ge N$,
- $f_n(K)\subset V$
- $\lVert f_n^{(i)}-f^{(i)}\rVert_K<\epsilon$ for $0\le i\le r$
Since the compact-open topology is coarser than this topology, convergence in the latter implies convergence in the former. So we can get rid of (1) and instead say that $f_n\xrightarrow{\operatorname{Diff}}f$ is equivalent to
- $f_n\xrightarrow{M}f$
- $f_n^{(i)}\xrightarrow{K}f^{(i)}$ for all $i$ and all valid $K$
[I'm writing $a\xrightarrow{L} b$ to mean uniform convergence on the compact set $L$. I'm writing $a\xrightarrow{\operatorname{Diff}}b$ to mean convergence in the topology on $\operatorname{Diff}(M)$.]
Finally, since we already know inversion and composition are continuous in the compact-open topology, we can focus on (4).
Now for inversion, suppose $f_n\xrightarrow{\operatorname{Diff}}f$. Then using the fact that $f^{-1}\circ f(x)=x$ and applying the chain rule repeatedly, we see that we can write
$$ f^{-(r)}\circ f(x)=c_r(f^{(1)}(x), f^{(2)}(x), \ldots, f^{(r)}(x))$$
where $c_r:\mathbb{R}^+\times\mathbb{R}^{r-1}\rightarrow\mathbb{R}$ is continuous. By choosing $n$ large enough, we can restrict the domain of $c_r$ [essentially by (1) and/or (3) above], and we can then assume $c_r$ is uniformly continuous. Then (4) plus the above equation implies
$$ f_n^{-(r)}\circ f_n\xrightarrow{K}f^{-(r)}\circ f$$
Finally, (3) implies we also have
$$ f_n^{-(r)}\circ f_n\xrightarrow{K}f_n^{-(r)}\circ f$$
and the fact that $f$ is a diffeomorphism shows we reallly have
$$ f_n^{-(r)}\xrightarrow{K}f^{-(r)}$$
which is enough for (4), and thus inversion is continuous.
For composition, again the chain rule applied to $g\circ f$ gives an equation like
$$ (g\circ f)^{(r)}(x) = d_r(g^{(1)}\circ f(x), \ldots, g^{(r)}\circ f, f^{(1)}(x), \ldots, f^{(r)}(x))$$
where we can assume $d_r$ is uniformly continuous.
Then (4) applied to $g$ shows we have
$$ (g_n\circ f)^{(r)}\xrightarrow{K}(g\circ f)^{(r)} $$
and (4) applied to $f$ then gives
$$ (g_n\circ f_n)^{(r)}\xrightarrow{K}(g\circ f)^{(r)} $$
The real insight here is that using the "convergent sequence" definition of continuity really simplifies the notation and presentation, and then really it all falls on the chain rule.
For surjectivity note that any continuous injection $\mathbb R\to\mathbb R$ is strictly monotonic. Since there is a strictly monotonic homeomorphism $(0,1)\cong \mathbb R$ (e.g. using tangent), the same is true for $(0,1)$.
Hence, a homeomorphism $g\colon (0,1)\to (0,1)$ must satisfy
$$
\lim_{x\to 1} g(x)=1 \quad\text{and}\quad \lim_{x\to 0} g(x) = 0
$$
when it is monotonically increasing, or
$$
\lim_{x\to 1} g(x)=0 \quad\text{and}\quad \lim_{x\to 0} g(x) = 1
$$
when it is monotonically decreasing.
In both cases, we obtain an extension of $g$ to a continuous bijection $[0,1]\to[0,1]$. Since the same is true for $g^{-1}$, the extension is a homeomorphism.
Best Answer
Here you don't need much topology, it boils down to pure manipulation of sets and bijective functions, and the two following facts: $(*)$ closed subsets of compact spaces are compact, and $(**)$ compact subsets of Hausdorff spaces are closed. Just take complements and use the fact that $h$ is by definition a bijection, so $$\begin{array}{RCL} h\in S(C,U) & \Longleftrightarrow & h(C)\subset U\\ & \Longleftrightarrow & X\setminus U\subset \underbrace{X\setminus h(C)}_{=h(X\setminus C)}\\ & \Longleftrightarrow & h^{-1}(X\setminus U)\subset X\setminus C \\ & \Longleftrightarrow & h^{-1}\in S(X\setminus U,X\setminus C) \end{array}$$ (which is a subbasic open neighborhood) i.e. $$\mathrm{inv}^{-1}( S(C,U))=\mathrm{inv}(S(C,U))=S(X\setminus U,X\setminus C)$$ which proves the inversion map $\mathrm{inv}$ to be continuous.