Let $U_1,U_2\subseteq\Bbb R^n$ be two invariant subspaces w.r.t. some group $\Gamma\subseteq\mathrm O(\Bbb R^n)$ of orthogonal matrices.
I wonder the following:
Question: If $U_1$ and $U_2$ are not orthogonal (i.e. there are $u_i\in U_i$ with $\langle u_1,u_2\rangle \not=0$), then every $T\in\Gamma$ restricted to $U_1\oplus U_2$ is already either $\mathrm{Id}$ or $-\mathrm{Id}$.
If this is true, what are the weakest assumptions we need? Do we need that the $U_i$ are irreducible? Does $U_1\cap U_2 = \{0\}$ sussfice, or maybe just $U_1\not\subseteq U_2$ and $U_2\not\subseteq U_1$.
Update
I received some very helpful answers and the statement as written above is definitely wrong. I still wonder whether there is a counterexample under the following restrictions:
- $\Gamma\subseteq\mathrm{O}(\Bbb R^n)$ is a finite group.
- The subspaces $U_i$ are irreducible invariant subspaces of $\Gamma$.
Best Answer
Take the natural action of $O_2(\mathbb R)$ on the space of two-by-two matrices by left multiplication. This space has an invariant inner product $\langle M,N\rangle = \operatorname{tr}(MN^T).$ Take
$$U_1=\Big\{\begin{pmatrix}a&0\\b&0\end{pmatrix}\mid a,b\in\mathbb R\Big\}$$ $$U_2=\Big\{\begin{pmatrix}a&a\\b&b\end{pmatrix}\mid a,b\in\mathbb R\Big\}.$$
These spaces are not orthogonal - set $a=b=1$ in both.