$\newcommand{\G}{\mathcal{G}}
\newcommand{\K}{\mathcal{K}} $Question: When does $ \G $ admit a finite maximal closed subgroup?
Answer : Must be one of the following two cases
- $ \G $ is compact and simple
- $ \G $ is not compact in which case $ \G $ cannot be connected and moreover the component group $ \G/\G^\circ $ does not preserve any nontrivial proper closed subgroup (see comment from YCor about $ C_5 \ltimes \mathbb{R}^2 $).
From now on I will confine myself to the case that $ \G $ is connected.
In other words I will consider the statement "A connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ if and only if $ \G $ is compact and simple."
The first implication is true.
Claim 1:
If a connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ then $ \G $ must be compact and simple.
Proof:
Let $ \G $ be a connected Lie group and $ G $ a finite maximal closed subgroup. Since $ G $ is finite then $ G $ is a compact subgroup of $ \G $ so must be contained in a maximal compact subgroup, call it $ \K $. But $ G $ is a maximal closed subgroup thus we must have that $ \K=\G $ (note that $ \K $ cannot equal $ G $ since $ \K $ is connected (the maximal compact of a connected group is always connected)). So $ \G $ must be compact. If $ \G $ is not simple then there exists some morphism
$$
\pi: \G \to \G_i
$$
with positive dimensional kernel (here $ \G_i $ is basically one of the semisimple factors of $ \G $). Then
$$
\pi^{-1}(\pi(G))
$$
is a closed positive dimensional subgroup containing $ G $, contradicting the fact that $ G $ is a finite maximal closed subgroup. Thus if a connected Lie group $ \G $ has a finite maximal closed subgroup then we can conclude that $ \G $ is simple.
However the reverse implication does not hold: $ SU_{15} $ is an example of a compact connected simple Lie group with no finite maximal closed subgroups.
To see why this is the case it is important to note that
Claim 2: For a compact connected simple Lie group $ \G $, $ G $ is a finite maximal closed subgroup of $ \G $ if and only if $ G $ is Ad-irreducible and $ G $ is a maximal finite subgroup of $ \G $.
this follows from Corollary 3.5 of Sawicki and Karnas - Universality of single qudit gates.
Since a finite subgroup of $ SU_n $ is Ad-irreducible if and only if it is a unitary 2-design we have
Claim 3: $ G $ is a finite maximal closed subgroup of $ SU_n $ if and only if $ G $ is a maximal unitary 2-group in $ SU_n $.
By inspecting Theorem 3 of Bannai, Navarro, Rizo, and Pham Huu Tiep - Unitary $t$-groups one immediately determines that $ SU_{15} $ has no finite maximal closed subgroups.
Some of the main examples of finite maximal closed subgroups of $ SU_n $ include the normalizer in $ SU_{p^n} $ of an extra-special group $ p^{2n+1} $. Here $ p $ is an odd prime. There is also a similar construction $ p=2 $. These are known as (complex) Clifford groups. Then there are infinite families of examples relating to the Weil module for $ \operatorname{PSp}_{2n}(3) $ and another family related to $ U_n(2) $. Plus many exceptional cases.
A similar normalizer construction to the above gives finite maximal closed subgroups of all the $ \operatorname{SO}(2^n) $ as normalizers of an extra-special group $ 2^{2n+1} $. This is known as the real Clifford group. For details about real and complex Clifford groups see Nebe, Rains, and Sloane - Self-Dual Codes and Invariant Theory.
(Comment converted to answer per request:)
The “surprising result” about simply connected homogeneous spaces in your (currently) last paragraph is Montgomery's Theorem (1950): generally (in your notation) if $G/H$ is compact and $H$ closed connected, then $K$ is transitive on $G/H$. The theorem is also discussed in Samelson (1952, p. 17).
Best Answer
$\DeclareMathOperator\SO{SO}\DeclareMathOperator\U{U}\DeclareMathOperator\O{O}$Denote by $\U'(n)$ the normalizer of $\U(n)$ in $\mathrm{GL}_{2n}(\mathbf{R})$. It is not hard to see that $\U(n)$ has index 2 in $\U'(n)$, which is generated by $\U(n)$ and by the coordinate-wise complex conjugation. Moreover, $\U'(n)$ is maximal in $\O(2n)$ (if $n\ge 2$).
In $G=\SO(5)$, consider the subgroup $H=\SO(5)\cap (\U'(2)\times \O(1))$ (which contains $\U(2)$ with index 2). I claim that $H$ is self-normalized, but not maximal, in $G=\SO(5)$.
The subgroup $H$ is not maximal in $G$ because it is properly contained in $L=SO(5)\cap (\O(4)\times \O(1))$.
It is self-normalized. Indeed, its action on $\mathbf{R}^5$ has the irreducible decomposition $4\oplus 1$, which is preserved by the normalizer. Hence, the normalizer of $H$ in $G$ equals the normalizer of $H$ inside $\SO(5)\cap (\O(4)\times \O(1))=L$. Using that $\U'(2)$ is maximal in $\O(4)$ one can deduce easily that $H$ is maximal in $L$. Since $H$ is not normal in $L$, it is self-normalized in $L$. Hence $H$ is self-normalized in $G$.