Question: Suppose $K$ is a compact subgroup of $GL(n,C)$ and $H$ is a Hilbert space. ($\pi$,$H$) is a unitary representation of $K$. Let $H_1$ and $H_2$ be two irreducible invariant subspace of $H$. Then $H_1$ and $H_2$ are orthogonal if $\pi$ restrict on $H_1$ and $H_2$ are nonisomorphic.
Here is my attempt: Let $y_i\in H_i$ fixed. Put $T: H_1\mapsto H_2; x\mapsto \int_K \langle x,\pi(\kappa)y_1\rangle\cdot\pi(\kappa)y_2d\kappa$. One can easily check $T$ is $K$-invariant. By Schur lemma, if $T$ is not isomorphism, then $T=0$. Thus $\langle Tx_1,x_2\rangle=0$ for every $x_i\in H_i$. This is equivalent to say that every matrix coefficient of $(\pi,H_1)$ is orthogonal in $L^2(K)$ to every matrix coefficient of $(\pi,H_2)$. But how can I say $H_1$ and $H_2$ themselves are orthogonal? Or can we chose $y_i$ to make sure $T$ is not 0? If we can, Schur lemma will implies $T$ is a isomorphism. Or maybe we can use peter-weyl theorem?
Any help or advice will be appropriated, thanks!
Best Answer
Your representation is probably continuous. If it is not then your integrals don't have to be well-defined. Moreover, your $T$ satisfies $g.T(x)=T(g.x)$, not only $g.T(x)\in T(H_1)$.
For $y\in H_2,x\in H_1$, $$T_y(x)= \int_K \langle x,k.y\rangle k.ydk \in H_2$$ Since the $K$-action on $H_2$ is unitary $$T_y(g.x)=\int_K \langle g.x,k.y\rangle k.ydk=\int_K \langle x,g^{-1}k.y\rangle k.ydk=\int_K \langle x,k.y\rangle (gk).yd(gk)=g.T_y(x)$$
Since $H_1,H_2$ are irreducible, either $T_yH_1=H_2$, $\ker(T_y)=0$ and $T_y$ is an isomorphism $H_1\to H_2$, or $T_y H_1=0$.
Thus if $(K,H_1),(K,H_2)$ are non-isomorphic then $$0=\langle T_y(x),x\rangle=\int_K \langle x,k.y\rangle \langle k.y,x\rangle dk=\int_K |\langle x,k.y\rangle|^2 dk \implies \langle x,y\rangle=0$$