Let $M$ be a smooth manifold.
I heard there is a way to introduce a topology and a structure of infinite dimensional manifold (something like a Banach or a Frechet manifold) on $\text{diff}(M)$ making it a group whose multiplication (composition) has some degree of smoothness with respect to the above structure.
My question: Is there a way to prove there aren't any topology and smooth structure making $\text{diff}(M)$ a finite dimensional Lie group w.r.t composition?
A possible strategy:
Finding topological obstructions.
I guess both topologies which are usually used (weak and strong) are not locally-compact, hence cannot serve as a ground for a finite dimensional smooth manifold.
But what about other possible topologies?
As noted by Matt it is possible to endow $\text{diff}(M)$ with a locally-compact, locally-connected and Hausdorff topology making it a topological group (The discrete topology).
Is it possible to endow it with locally-compact and locally-connected Hausdorff second countable topology making it a topological group whose action on $M$ is continuous?
(All these properties are necessary for the topology of a Lie group, hence if we show this is impossible, we are done).
Best Answer
For sake of completeness I am writing a solution based on the suggestion of Mike Miller:
Theorem: There are no topology $\tau$ and a compatible (finite-dimensional) smooth structure $A$ on $\text{Diff(M)}$ satisfying:
(1) The action of $\text{Diff(M)}$ on $M$ is continuous w.r.t $\tau$.
(2) $\text{Diff(M)}$ is a Lie group w.r.t $(\tau,A)$.
Proof:
We assume by contradiction there exist such a pair $(\tau,A)$.
Define $X_n = \{(p_1,...p_n)\in M^n |p_i \neq p_j \forall i \neq j \}$, and look at the map $\psi : \text{Diff(M)} \times X_n \rightarrow X_n$ , $\psi\left(\phi,(p_1,...p_n)\right)=(\phi(p_1),...\phi(p_n))$.
(1) It is easy to see that $X_n$ is an $n$-dimensional manifold. (It's an open subset of $M^n$).
(2) Continuity of the action of $\text{Diff(M)}$ on $M$ implies $\psi$ is continuous.
(3) n-transitivity of $\text{Diff(M)}$ implies the action $\psi$ (of $\text{Diff(M)}$ on $X_n$) is transitive.
Now fix some point $q=(q_1,...q_n)\in X_n$, and denote by $G_q = \{\phi \in \text{Diff(M)} | \phi(q)=q \} $ the stabilizer group. It is closed, hence by the closed subgroup theorem $G_q$ is an embedded Lie subgroup of $\text{Diff(M)}$.
So, The left coset space $\text{Diff(M)} / G_q$ is a topological manifold* of dimension $dim(\text{Diff(M)})-dim(G_q)$.
(It can also be given a unique smooth structure making the quotient map $\pi: \text{Diff(M)} \rightarrow \text{Diff(M)} / G_q$ a smooth submersion, but that is irrelevant for our discussion**).
Now , by easy proposition on continuous homogenous $G$-spaces (i.e topological spaces with a continuous transitive action of a topological group $G$) it follows that $\text{Diff(M)} / G_q$ and $X_n$ are homeomorphic.
In particular their dimension as topological manifolds are equal, hence: $dim(\text{Diff(M)}) \ge dim(\text{Diff(M)})-dim(G_q) = dim(X_n)=n$ for every $n$ which is a contradiction.
*See Lee's book (Intro to smooth manifolds) Thm 21.17.
** If we assume the action of $\text{Diff(M)}$ on $M$ is smooth (not merely continuous), then we get that $\text{Diff(M)} / G_q$ and $X_n$ are diffeomorphic. (See Thm 21.18)