Non-orthogonal invariant subspaces

group-theoryinvariant-subspacelinear algebramatricesrepresentation-theory

Let $\Gamma\subset\mathrm O(\Bbb R^n)$ be a finite group of orthogonal matrices. Let $U_1,U_2\subseteq\Bbb R^n$ be two irreducible invariant subspaces w.r.t. $\Gamma$ with $U_1\cap U_2=\{0\}$, which are not orthogonal to each other, i.e. there are $u_i\in U_i$ with $\langle u_1,u_2\rangle \not=0$.

I was sceptic about the existence of such, but you can find examples here in a previous question of mine. Thinking a bit about such subspaces, I came to the following question:

Question: Is it true, that:

  1. $\dim U_1=\dim U_2=:d$.
  2. Every other $d$-dimensional subspace $U\subset U_1\oplus U_2$ with $U\cap U_i=\{0\}$ is an irreducible invariant subspace as well.
  3. There are two orthogonal $d$-dimensional irreducible invariant subspaces $\bar U_1,\bar U_2\subset U_1\oplus U_2$.

Update

The second statement is not correct, but should be substituted by a different one. One version was given in the answer of Joppy. I can also think about something like this: every $u\in U_1\oplus U_2\setminus\{0\}$ is contained in exactly one $d$-dimensional irreducible invariant subspace $U\subset U_1\oplus U_2$.

Best Answer

Allow me to translate this into more common representation-theoretic language. Saying that you have a finite subgroup $\Gamma \subseteq O(\mathbb{R}^n)$ is the same as the following data:

  1. A finite group $G$,
  2. A finite-dimensional real vector space $V$ equipped with a representation $\rho: G \to \operatorname{GL}(V)$, and
  3. An inner product $\langle -, - \rangle: V \times V \to \mathbb{R}$ which is $G$-invariant, in the sense that $\langle \rho(g) v, \rho(g) u \rangle = \langle v, u \rangle$ for all $g \in G$ and $u, v \in V$.

Let $\{I_\lambda \mid \lambda \in \Lambda\}$ be a complete set of irreducible real representations of $G$. Simply knowing that $V$ is a real representation means that there is a canonical decomposition of $V$ into isotypic components, $V = \bigoplus_{\lambda} V_\lambda$, where $\lambda$ ranges over some indexing set for the isomorphism classes of irreducible representations of $G$. Here the subspace $V_\lambda$ is defined as the sum of all subrepresentations of $V$ isomorphic to $I_\lambda$. What is interesting is that these $V_\lambda$ must be orthogonal to each other.

Lemma: Suppose that $U, W \subseteq V$ are irreducible representations, and $\langle U, V \rangle \neq 0$. Then $U \cong V$ as real representations.

Proof: Since $\langle U, V \rangle \neq 0$, the map $\phi: U \to V^*, \phi(u)(v) = \langle u, v \rangle$ is nonzero. Furthermore, the $G$-invariancy of the inner product ensures that $\phi$ is a map of representations. Since $V \cong V^*$ as representations, we have found a nonzero $G$-equivariant map $U \to V$. By Schur's lemma, $U \cong V$.

This lemma shows that all the isotypic components $V_\lambda$ must be orthogonal under the $G$-invariant inner product. The answers to the rest of your questions basically follow from knowing that decomposition:

Answers to questions: let $U_1, U_2$ be irreducible subrepresentations of $V$, such that $\langle U_1, U_2 \rangle \neq 0$. Then:

  1. $\dim U_1 = \dim U_2$, since by the above they must be isomorphic representations.
  2. Every other nonzero $G$-invariant ($\dim U_1$)-dimensional subspace of $U_1 \oplus U_2$ must be isomorphic to $U_1$ as a representation, and hence irreducible. ($U_1 \oplus U_2$ is still inside the isotypic component).
  3. The orthogonal complement of $U_1$ inside $U_1 \oplus U_2$ will be a subrepresentation which is both isomorpic and orthogonal to $U_1$.
Related Question