$\newcommand{\G}{\mathcal{G}}
\newcommand{\K}{\mathcal{K}} $Question: When does $ \G $ admit a finite maximal closed subgroup?
Answer : Must be one of the following two cases
- $ \G $ is compact and simple
- $ \G $ is not compact in which case $ \G $ cannot be connected and moreover the component group $ \G/\G^\circ $ does not preserve any nontrivial proper closed subgroup (see comment from YCor about $ C_5 \ltimes \mathbb{R}^2 $).
From now on I will confine myself to the case that $ \G $ is connected.
In other words I will consider the statement "A connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ if and only if $ \G $ is compact and simple."
The first implication is true.
Claim 1:
If a connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ then $ \G $ must be compact and simple.
Proof:
Let $ \G $ be a connected Lie group and $ G $ a finite maximal closed subgroup. Since $ G $ is finite then $ G $ is a compact subgroup of $ \G $ so must be contained in a maximal compact subgroup, call it $ \K $. But $ G $ is a maximal closed subgroup thus we must have that $ \K=\G $ (note that $ \K $ cannot equal $ G $ since $ \K $ is connected (the maximal compact of a connected group is always connected)). So $ \G $ must be compact. If $ \G $ is not simple then there exists some morphism
$$
\pi: \G \to \G_i
$$
with positive dimensional kernel (here $ \G_i $ is basically one of the semisimple factors of $ \G $). Then
$$
\pi^{-1}(\pi(G))
$$
is a closed positive dimensional subgroup containing $ G $, contradicting the fact that $ G $ is a finite maximal closed subgroup. Thus if a connected Lie group $ \G $ has a finite maximal closed subgroup then we can conclude that $ \G $ is simple.
However the reverse implication does not hold: $ SU_{15} $ is an example of a compact connected simple Lie group with no finite maximal closed subgroups.
To see why this is the case it is important to note that
Claim 2: For a compact connected simple Lie group $ \G $, $ G $ is a finite maximal closed subgroup of $ \G $ if and only if $ G $ is Ad-irreducible and $ G $ is a maximal finite subgroup of $ \G $.
this follows from Corollary 3.5 of Sawicki and Karnas - Universality of single qudit gates.
Since a finite subgroup of $ SU_n $ is Ad-irreducible if and only if it is a unitary 2-design we have
Claim 3: $ G $ is a finite maximal closed subgroup of $ SU_n $ if and only if $ G $ is a maximal unitary 2-group in $ SU_n $.
By inspecting Theorem 3 of Bannai, Navarro, Rizo, and Pham Huu Tiep - Unitary $t$-groups one immediately determines that $ SU_{15} $ has no finite maximal closed subgroups.
Some of the main examples of finite maximal closed subgroups of $ SU_n $ include the normalizer in $ SU_{p^n} $ of an extra-special group $ p^{2n+1} $. Here $ p $ is an odd prime. There is also a similar construction $ p=2 $. These are known as (complex) Clifford groups. Then there are infinite families of examples relating to the Weil module for $ \operatorname{PSp}_{2n}(3) $ and another family related to $ U_n(2) $. Plus many exceptional cases.
A similar normalizer construction to the above gives finite maximal closed subgroups of all the $ \operatorname{SO}(2^n) $ as normalizers of an extra-special group $ 2^{2n+1} $. This is known as the real Clifford group. For details about real and complex Clifford groups see Nebe, Rains, and Sloane - Self-Dual Codes and Invariant Theory.
$\DeclareMathOperator\SO{SO}\DeclareMathOperator\U{U}\DeclareMathOperator\O{O}$Denote by $\U'(n)$ the normalizer of $\U(n)$ in $\mathrm{GL}_{2n}(\mathbf{R})$. It is not hard to see that $\U(n)$ has index 2 in $\U'(n)$, which is generated by $\U(n)$ and by the coordinate-wise complex conjugation. Moreover, $\U'(n)$ is maximal in $\O(2n)$ (if $n\ge 2$).
In $G=\SO(5)$, consider the subgroup $H=\SO(5)\cap (\U'(2)\times \O(1))$ (which contains $\U(2)$ with index 2). I claim that $H$ is self-normalized, but not maximal, in $G=\SO(5)$.
The subgroup $H$ is not maximal in $G$ because it is properly contained in $L=SO(5)\cap (\O(4)\times \O(1))$.
It is self-normalized. Indeed, its action on $\mathbf{R}^5$ has the irreducible decomposition $4\oplus 1$, which is preserved by the normalizer. Hence, the normalizer of $H$ in $G$ equals the normalizer of $H$ inside $\SO(5)\cap (\O(4)\times \O(1))=L$. Using that $\U'(2)$ is maximal in $\O(4)$ one can deduce easily that $H$ is maximal in $L$. Since $H$ is not normal in $L$, it is self-normalized in $L$. Hence $H$ is self-normalized in $G$.
Best Answer
Yes, it is true. Let us show that, in a compact Lie group $G$ with $G^0$ nonabelian, finite subgroups that are not contained in any proper subgroup of positive dimension have bounded cardinal.
By contradiction, let finite subgroups $G_n$ have cardinal tending to infinity, and not contained in any proper subgroup of positive dimension.
Let $B$ be a neighborhood of 0 in the Lie algebra of $G$ such that $\exp|_B$ is a homeomorphism onto its image, with inverse denoted by $\log$ (we will let $B$ vary, but can assume it is contained in a single such neighborhood $B_0$ once and for all, hence the function $\log$ will not depend on $B$).
Define $T_n^B=\exp\big(\mathrm{span}\big(\log(G_n\cap\exp(B))\big)\big)$.
The first point is that for each $B\subset B_0$ and for $n$ large enough, $G_n\cap\exp(B)$ consists of commuting elements. This is close to Jordan's theorem and is a particular case of Zassenhaus' theorem (see for instance Theorem 1.3 here, with in mind that connected nilpotent subgroups of compact groups are abelian). We can extract once and for all to suppose that this holds for all $n$.
The next point is that $T_n^B$ is a torus. For an arbitrary commutative subset, it might have been dense in a torus. But the point is that elements of finite subgroups are well-behaved. More precisely: for this point, we can suppose that $G$ is embedded in the unitary group $\mathrm{U}(d)$. Every element of $G$ can be diagonalized with eigenvalues $e^{i\pi k_1/m},\dots,e^{i\pi k_d/m}$. We can arrange $B_0$ so that it avoids elements with eigenvalues $-1$ and define the log accordingly, so assume all $k_i$ to be in $]-m,m[$. Hence the log of such an element will be, in the same diagonal basis, the matrix with eigenvalues $i\pi k_1/m,\dots i\pi k_d/m$, and the exp of its span will be the 1-dimensional torus of diagonal matrices with eigenvalues $\theta^{k_1},\dots,\theta^{k_d}$ for $\theta$ in the unit circle. Now when we consider all $G_n\cap \exp(B)$, we just consider the torus generated by several such (commuting) tori.
Now consider $\liminf_n \dim(T_n^B)$. This is positive (because $G_n$ is infinite, so that $G_n\cap\exp(B)$ is never reduced to $\{1\}$. This liming decreases when $B$ decreases, and hence we can find $B_1$ for which its value, say $c>0$, is minimal. After extracting once again, we an suppose that $\lim_n\dim(T_n^{B_1})=c$. Define $T_n=T_n^{B_1}$. Hence, for every neighborhood $B\subset B_1$, there exists $n_0$ (depending on $B$) such that for all $n\ge n_0$ we have $T_n^B=T_n$ (because $T_n^B\subset T_n$ and these are tori of the same dimension).
This observation allows to deduce that $T_n$ is, for large $n$, normalized by $G_n$. Indeed, for $g_n\in G_n$, we have $$g_nT_n^Bg_n^{-1}=\exp(\mathrm{span}\log(G_n\cap \exp(g_nBg_n^{-1})));$$ since $1$ has a basis of conjugation-invariant neighborhoods, we can choose $B$ such that $\bigcup_g gBg^{-1}$ is contained in $B_1$. We deduce that $g_nT_ng_n^{-1}\subseteq T_n$ for large $n$, hence this is an equality.
Then $G_nT_n$ is a positive-dimensional closed subgroup containing $G_n$. By assumption, it equals $G$, and this is a contradiction since $G^0$ is non-abelian.