About 20 years ago I read in textbook that
"all irreducible representations of compact groups are finite-dimensional", but
me and the proof of this fact never met each other 🙂
May I ask dear MO colleagues, is there (simple?) argument to prove it ?
As far as I heard this result can be generalized in the realm of non-commutative geometry,
Woronowicz compact quantum groups (?).
So the "bonus" question – what is appropriate "compactness" condition for some algebra (and/or Hopf algebra) such that it will guarantee the same property
(i.e. all irreps are finite-dim.) ?
[EDIT] Thanks very much for excellent answers ! Let me ask about some more details, to finally clarify.
1) What is maximal possible relaxation of the requirement on vector space V ? Is it enough to require arbitrary
linear topological space or we need to restrict to Hausdorff (?) Banach (?) Hilbert (?), whatever spaces ? (It seems restrictions on the space may come from the Schur lemma, it is not clear for me what is appropriate generality it holds).
2) Do we need axiom of choice here ? (Probably not, we need existence of Haar measure, but Wikipedia writes that "Henri Cartan furnished a proof of existence of Haar measure which avoided AC use.[4]"
3) Informally: what is the hardest tool one uses in the proof ? (May be existence of Haar measure ?)
[END EDIT].
[EDIT]
Let me add sketch of arguments by Aakumadula, as I understand it. It might be helpful to clarify new questions.
1) Tool: Continuous functions on the group can be mapped to operators on V. (Need measure here). (Group algebra acts on V).
2) Fact: Continuous function will be mapped to COMPACT operators. (In R^n I know how to prove it, in general no).
3) Observe: Conjugation invariant function are mapped to operators which commute with action of group.
4) Schur Lemma: operators commuting with group in irrep are Lamda*Id. (What do we need from the space V for this to be true ? )
5) Corollary: If we find invariant continuous function which is mapped in NON-zero in V,
then we are done, because by (2) it is compact operator and by (4) it is Lambda*Id.
So we need to find invariant function which will be non-zero in V.
6) Take arbitrary "approximate identity" i.e. sequence of continuous (non-invariant) functions f_n which
converge as functionals to delta-function in identity of the group. (It is local fact. But how to prove it ? Do we need Axiom of choice here ? )
7) Make averaging over the group of f_n – get sequence of INVARIANT continuous functions which again converge to detla(e), since delta(e) is invariant.
8) Operators T(f_n) converge to identity operator, hence for some N they are NON-ZERO.
WE ARE DONE by (5) ! Because T(f_N) is compact and Lambda*Id and Lambda is NON-ZERO.
[End EDIT].
Best Answer
[First proof] I will address only the first part. Suppose $G$ is a compact group and $\pi : G \rightarrow U(H)$ a unitary irreducible representation on a Hilbert space $H$. Suppose $\phi$ a continuous function on $G$. Then it is easy to show that $\pi (\phi)$ is a compact operator on $H$.
Since the OP has asked for clarification let me state: convolution by continuous functions on $L^2(G)$ where $G$ a COMPACT group, is a compact operator. This fact is used in all proofs of Peter-Weyl. The proof is by showing that the convolution is an $L^2$ kernel, and any $L^2$ function on $G\times G$ may be approximated in $L^2$ by simple functions (linear combination of char functions of the form $E\times F$ where $E,F$ ae measurable subsets of $G$), each of these simple kernels have finite dimensional image and are hence compact operators. This is standard material in any functional analysis book (in fact Kirillov's book).
What I have used is that for a compact group, any irreducible unitary representation is a sub of the regular representation (see the comments), and hence $\pi (\phi)$ ,which is a convolution by $\phi$ is a compact operator.
If $(\phi _{\epsilon} )$ is an approximate identity on $G$ consisting of continuous conjugation invariant real valued continuous functions on $G$, then $\pi (\phi _{\epsilon})$ is a sequence of compact operators (which are scalars because of irreducibility) converging weakly to the identity, and hence the identity operator is also compact. Therefore, $H$ is finite dimensional.
I mention this because this does not use the Peter-Weyl theorem (but uses ideas of the proof).
[Second Proof] If you use the Peter-Weyl theorem, then the proof of finite dimensionality is easy. Peter weyl says that $L^2(G)$ is a Hilbert space direct sum of irreducible FINITE dimensional representations of $G$. The algebraic direct sum $X$ is then a dense subspace of $L^2(G)$. If an infinite dimensional irrep $H$ existed, it would still be in $L^2(G)$ but by Schur orthogonality (another use of Schur's lemma) functions in $H$ would be orthogonal to functions in $X$, a contradiction, since $X$ is dense.
[Remark] if $G$ is already a closed subgroup of $U(n)$, and $V$ is the standard rep of $U(n)$ then by the Stone-Weierstrass theorem, $V$, $V^*$ and irreducible subs of their tensor powers, will all be in $L^2(G)$ and will be dense in the space of continuous functions on $G$. And hence these representation functions $Y$ will be dense in $L^2(G)$. So vectors in $H$ cannot be orthogonal to the elements of $Y$. All this is standard material in any book on the Peter-Weyl theorem (actually worked out in Kirillov's book) and I suggest that the OP looks at them
[Remark] The proof given by Francois Ziegler is the simplest, in my opinion. .