This question is inspired by this post.
Question. Does the automorphism group of a (finite-dimensional real) Lie group $ G $ always have a (natural) Lie group structure?
Let $ \mathfrak{g} $ be the Lie algebra of $ G $.
To my understanding, an automorphism $ \phi\colon G \to G $ induces an automorphism $ d\phi\colon \mathfrak{g} \to \mathfrak{g} $. If $ G $ is connected, this correspondence $ \operatorname{Aut} G \to \operatorname{Aut} \mathfrak{g} $ is injective, so we consider $ \operatorname{Aut} G $ as a subgroup of $ \operatorname{Aut} \mathfrak{g} $, which is in turn a closed subgroup of $ \mathit{GL}(\mathfrak{g}) $. According to the above post, the answer is affirmative if $ G $ is simply connected; in this case, the correspondence $ \phi \mapsto d\phi $ is an isomorphism and hence $ \operatorname{Aut} G \cong \operatorname{Aut} \mathfrak{g} $ has a natural Lie group structure as a closed subgroup of $ \mathit{GL}(\mathfrak{g}) $ by Cartan’s theorem.
Is this still holds if we don’t assume that $ G $ is simply connected? If $ G $ is connected, we can restate the problem as follows thanks to Cartan’s theorem:
Question'. Is the image of the embedding $ \operatorname{Aut} G \to \operatorname{Aut} \mathfrak{g} $; $ \phi \mapsto d\phi $ always closed in $ \mathit{GL}(\mathfrak{g}) $?
Edit. Thanks to José Carlos Santos’ comment, I understood that the answer is affirmative if $ G $ is connected. Then, does $ \operatorname{Aut} G $ have a natural Lie group structure if $ G $ is not necessarily connected? In this case, we cannot consider $ \operatorname{Aut} G $ as a subgroup of $ \mathit{GL}(\mathfrak{g}) $, so I have no idea how to give $ \operatorname{Aut} G $ a Lie group structure (if possible).
Best Answer
I will write an answer assuming the “textbook” definition of a manifold (in particular, of a Lie group), i.e., that manifolds are required to satisfy the 2nd countability axiom (not everybody shares this view). Then the answer to your question is, in general, negative. Namely, consider the group $$ G=\bigoplus_{\mathbb N} {\mathbb Z}_3 $$ which is a countably-infinite direct sum of finite groups of order 3. Equipped with the discrete topology, this is a Lie group (since it is countable). At the same time, the automorphism group of $G$ contains a subgroup isomorphic to $$ A=\prod_{{\mathbb N}} {\mathbb Z}_2. $$ (The element $a\in A$ with components $a_i$ acts on the $i$-th factor of $G$ by the trivial automorphism if $a_i=0$ and the unique nontrivial automorphism if $a_i=1$.) The group $A$ has the cardinality of continuum. It is not hard to see that no subgroup of $A$ of countable index is isomorphic (as an abstract group) to a subgroup of $$ S^1\times \ldots \times S^1. $$ Hence, it is also not isomorphic to a subgroup of any compact Lie group. Hence, it is not isomorphic to any subgroup of a Lie group.
At the same time, as is noted in comments, if $G$ is a Lie group such that $G/G_0$ is finitely generarted, then $H=\operatorname{Aut}(G)/\operatorname{Aut}(G_0)$ is countable. Since $\operatorname{Aut}(G_0)$ has a natural structure of a Lie group, it follows that $\operatorname{Aut}(G)$ has one as well, namely, where its topology is of the disjoint union $$ \coprod_{H} h\cdot \operatorname{Aut}(G_0). $$