Remember that invariant subspaces of a representation $\pi:G\to GL(V)$ are the linear spans of sets of the form $\{ \pi(g) v\vert g\in G, v\in A\}$ for $A\subseteq V$. So a good starting point is studying the spans of orbits of points.
- You can check directly that the orbits of nonzero constant polynomials are just singletons, and they span the space of all constant polynomials;
- the orbits of linear polynomials are sets of the form $V'_a=\{\alpha(x+a+t)\vert t\in {\bf R}\}$ for some $a,\alpha\in {\bf C}$ with $\alpha\neq 0$, and it is easy to see that they span a space containing constants and from that it's easy to see that they all span the entire space of polynomials of degree $\leq 1$;
- similarly, you can show that the span of each orbit of a quadratic contains a linear polynomial and hence its orbit, so from that, all linear polynomials, and so also all quadratics, so it is the entire space.
Allow me to translate this into more common representation-theoretic language. Saying that you have a finite subgroup $\Gamma \subseteq O(\mathbb{R}^n)$ is the same as the following data:
- A finite group $G$,
- A finite-dimensional real vector space $V$ equipped with a representation $\rho: G \to \operatorname{GL}(V)$, and
- An inner product $\langle -, - \rangle: V \times V \to \mathbb{R}$ which is $G$-invariant, in the sense that $\langle \rho(g) v, \rho(g) u \rangle = \langle v, u \rangle$ for all $g \in G$ and $u, v \in V$.
Let $\{I_\lambda \mid \lambda \in \Lambda\}$ be a complete set of irreducible real representations of $G$. Simply knowing that $V$ is a real representation means that there is a canonical decomposition of $V$ into isotypic components, $V = \bigoplus_{\lambda} V_\lambda$, where $\lambda$ ranges over some indexing set for the isomorphism classes of irreducible representations of $G$. Here the subspace $V_\lambda$ is defined as the sum of all subrepresentations of $V$ isomorphic to $I_\lambda$. What is interesting is that these $V_\lambda$ must be orthogonal to each other.
Lemma: Suppose that $U, W \subseteq V$ are irreducible representations, and $\langle U, V \rangle \neq 0$. Then $U \cong V$ as real representations.
Proof: Since $\langle U, V \rangle \neq 0$, the map $\phi: U \to V^*, \phi(u)(v) = \langle u, v \rangle$ is nonzero. Furthermore, the $G$-invariancy of the inner product ensures that $\phi$ is a map of representations. Since $V \cong V^*$ as representations, we have found a nonzero $G$-equivariant map $U \to V$. By Schur's lemma, $U \cong V$.
This lemma shows that all the isotypic components $V_\lambda$ must be orthogonal under the $G$-invariant inner product. The answers to the rest of your questions basically follow from knowing that decomposition:
Answers to questions: let $U_1, U_2$ be irreducible subrepresentations of $V$, such that $\langle U_1, U_2 \rangle \neq 0$. Then:
- $\dim U_1 = \dim U_2$, since by the above they must be isomorphic representations.
- Every other nonzero $G$-invariant ($\dim U_1$)-dimensional subspace of $U_1 \oplus U_2$ must be isomorphic to $U_1$ as a representation, and hence irreducible. ($U_1 \oplus U_2$ is still inside the isotypic component).
- The orthogonal complement of $U_1$ inside $U_1 \oplus U_2$ will be a subrepresentation which is both isomorpic and orthogonal to $U_1$.
Best Answer
We assume that $\alpha\notin \pi\mathbb{Z}$. Your matrix, say $R$, has $3$ distinct eigenvalues over $\mathbb{C}$: $e^{i\alpha},e^{-i\alpha},1$ with associated eigenvectors $v,\bar{v},e_3$. A general result says that the proper invariant subspaces over $\mathbb{C}$ are the $span(U)$, where $U$ goes through the strict subsets of $\{v,\bar{v},e_3\}$ (there are $6$ such vector spaces).
Over $\mathbb{R}$, you must group the two first eigenvectors and you cannot group $v,e_3$ or $\bar{v},e_3$. Then, there are only two proper invariant spaces: $span_{\mathbb{R}}(e_1,e_2)\subset span_{\mathbb{C}}(v,\bar{v})$ and $span(e_3)$.