Consider a continuous irreducible representation of a compact Lie group on a finite-dimensional complex Hilbert space. There are three mutually exclusive options:
1) it's not isomorphic to its dual (in which case we call it 'complex')
2) it has a nondegenerate symmetric bilinear form (in which case we call it 'real')
3) it has a nondegenerate antisymmetric bilinear form (in which case we call it 'quaternionic')
It's 'real' in this sense iff it's the complexification of a representation on a real vector space, and it's 'quaternionic' in this sense iff it's the underlying complex representation of a representation on a quaternionic vector space.
Offhand, I know just four compact Lie groups whose continuous irreducible representations on complex vector spaces are all either real or quaternionic in the above sense:
1) the group Z/2
2) the trivial group
3) the group SU(2)
4) the group SO(3)
Note that I'm so desperate for examples that I'm including 0-dimensional compact Lie groups, i.e. finite groups!
1) is the group of unit-norm real numbers, 2) is a group covered by that, 3) is the group of unit-norm quaternions, and 4) is a group covered by that. This probably explains why these are all the examples I know. For 1), 2) and 4), all the continuous irreducible representations are in fact real.
What are all the examples?
Best Answer
An irreducible representation is real or quaternionic precisely when its character is real-valued. By the Peter-Weyl theorem all characters are real-valued precisely when every element in the group is conjugate to its inverse. When the group is connected a more precise answer is as follows: The Weyl group (in its tautological representation) must contain multiplication by $-1$ and this is true precisely when all indecomposable root system factors have that property. I don't remember off hand which indecomposable root systems have this property but it is of course well known (type A is out, type B/C is in, type D depends on the parity of the rank).
Addendum: Found the relevant places in Bourbaki. All characters are real-valued precisely when the element he calls $w_0$ is $-1$ (Ch. VIII,Prop. 7.5.11) and one can also read off if a given representation is real or quaternionic (loc. cit. Prop 12). From the tables in Chapter 6 one gets that $w_0=-1$ precisely for $A_1$, B/C, D for even rank, $E_7$, $E_8$, $F_4$ and $G_2$.