$\newcommand{\G}{\mathcal{G}}
\newcommand{\K}{\mathcal{K}} $Question: When does $ \G $ admit a finite maximal closed subgroup?
Answer : Must be one of the following two cases
- $ \G $ is compact and simple
- $ \G $ is not compact in which case $ \G $ cannot be connected and moreover the component group $ \G/\G^\circ $ does not preserve any nontrivial proper closed subgroup (see comment from YCor about $ C_5 \ltimes \mathbb{R}^2 $).
From now on I will confine myself to the case that $ \G $ is connected.
In other words I will consider the statement "A connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ if and only if $ \G $ is compact and simple."
The first implication is true.
Claim 1:
If a connected Lie group $ \G $ has a finite maximal closed subgroup $ G $ then $ \G $ must be compact and simple.
Proof:
Let $ \G $ be a connected Lie group and $ G $ a finite maximal closed subgroup. Since $ G $ is finite then $ G $ is a compact subgroup of $ \G $ so must be contained in a maximal compact subgroup, call it $ \K $. But $ G $ is a maximal closed subgroup thus we must have that $ \K=\G $ (note that $ \K $ cannot equal $ G $ since $ \K $ is connected (the maximal compact of a connected group is always connected)). So $ \G $ must be compact. If $ \G $ is not simple then there exists some morphism
$$
\pi: \G \to \G_i
$$
with positive dimensional kernel (here $ \G_i $ is basically one of the semisimple factors of $ \G $). Then
$$
\pi^{-1}(\pi(G))
$$
is a closed positive dimensional subgroup containing $ G $, contradicting the fact that $ G $ is a finite maximal closed subgroup. Thus if a connected Lie group $ \G $ has a finite maximal closed subgroup then we can conclude that $ \G $ is simple.
However the reverse implication does not hold: $ SU_{15} $ is an example of a compact connected simple Lie group with no finite maximal closed subgroups.
To see why this is the case it is important to note that
Claim 2: For a compact connected simple Lie group $ \G $, $ G $ is a finite maximal closed subgroup of $ \G $ if and only if $ G $ is Ad-irreducible and $ G $ is a maximal finite subgroup of $ \G $.
this follows from Corollary 3.5 of Sawicki and Karnas - Universality of single qudit gates.
Since a finite subgroup of $ SU_n $ is Ad-irreducible if and only if it is a unitary 2-design we have
Claim 3: $ G $ is a finite maximal closed subgroup of $ SU_n $ if and only if $ G $ is a maximal unitary 2-group in $ SU_n $.
By inspecting Theorem 3 of Bannai, Navarro, Rizo, and Pham Huu Tiep - Unitary $t$-groups one immediately determines that $ SU_{15} $ has no finite maximal closed subgroups.
Some of the main examples of finite maximal closed subgroups of $ SU_n $ include the normalizer in $ SU_{p^n} $ of an extra-special group $ p^{2n+1} $. Here $ p $ is an odd prime. There is also a similar construction $ p=2 $. These are known as (complex) Clifford groups. Then there are infinite families of examples relating to the Weil module for $ \operatorname{PSp}_{2n}(3) $ and another family related to $ U_n(2) $. Plus many exceptional cases.
A similar normalizer construction to the above gives finite maximal closed subgroups of all the $ \operatorname{SO}(2^n) $ as normalizers of an extra-special group $ 2^{2n+1} $. This is known as the real Clifford group. For details about real and complex Clifford groups see Nebe, Rains, and Sloane - Self-Dual Codes and Invariant Theory.
One can consult the tables of Bray-Holt-Roney Dougal to work out the subgroups of $\mathrm{PSU}_m$.
- For $m=5$, we have copies of $PSL_2(11)$ and $PSU_4(4)$.
- For $m=6$ we have $PSL_2(11)$, $PSL_3(4)$ and $PSU_4(9)$.
- For $m=7$ we have $PSU_3(9)$.
- For $m=8$ we have $PSL_3(4)$.
- For $m=9$ we have $PSL_2(19)$.
- For $m=10$ we have $PSL_2(19)$ and $PSL_3(4)$.
- For $m=11$ we have $PSL_2(23)$ and $PSU_5(4)$.
- For $m=12$ we have $PSL_2(23)$ and $PSL_2(9)$.
So far, so good. Bray-Holt-Roney Dougal stops at dimension 12. For dimensions 13 to 15 one consults tables in Anna Schroeder's PhD thesis (St Andrews, 2015) and there are examples in those dimensions. For dimensions 16 and 17 one consults the thesis of Daniel Rogers (Warwick, 2017). For 16 there is an example, but there are no interesting subgroups (i.e., finite subgroups not contained in a closed positive-dimensional subgroup) of $\mathrm{PSU}_{17}$ at all, never mind linear or unitary ones.
So it seems $m=17$ is the first case where things go wrong.
Best Answer
The comment of @abx is part of a general picture. For $n >7$ the alternating group $A_{n}$ has no non-trivial complex irreducible character of degree less than $n-1$, so that for $n > 7$, $A_{n}$ is not isomorphic to any subgroup of ${\rm GL}(n-3, \mathbb{C}).$
Furthermore, if $n > 7$ is also even, then the double cover of $A_{n}$ (which, by a theorem of I. Schur is a maximal perfect central extension of $A_{n}$) has no faithful complex irreducible character of odd degree, so certainly not of degree $n-3$ ( as an involution in its centre would be represented by a matrix of determinant $-1$). Hence the double cover of $A_{n}$ has no non-trivial irreducible complex representation of degree $n-3$, faithful or not. Thus $A_{n}$ does not embed as an irreducible subgroup of ${\rm PSU}(n-3, \mathbb{C})$ (and it is clear that a maximal closed subgroup has to be irreducible).
(A few points for clarity: for finite groups, all finite dimensional complex representations are equivalent to unitary ones. Also, there is no real distinction between projective representations (in Schur's sense) and genuine representations of covering groups. The fact that the minimal degree of an non-triival irreducible complex representation of $A_{n}$ is $n-1$ ( for $n > 7$) can be found in almost any text on representation theory of $S_{n}$ ( eg that of G.D. James)).
Later edit: I am not sure of the smallest degree of a faithful projective (in Schur's sense) complex representation of $A_{n}$, though I believe this must be well-known to symmetric group experts. However, results such as Zsygmondy's theorem provide a lower bound. If $p$ is a prime less than or equal to $m-2$, then $A_{m}$ contains a $p$-cycle which is conjugate to all its non-identity powers, from which it follows that $A_{m}$ can have no non-trivial complex projective representation of degree less than $p-1.$ Hence if $n$ is an even integer such that there is a prime $p$ with $\frac{n}{2} < p < n-2$, we see that $A_{n}$ has no complex irreducible projective representation of degree less than $\frac{n}{2}$ (and we may find similar bounds for odd $n$).