I am a french-speaking undergraduate physics student. Therefore, I do not know of the specific English terminologies involved in the following discussion. I have tried to check everything up, but I might have done some mistakes. That's why I am first going to define everything useful to my question.
Homomorphism. For any linear map $T$ from $X$ to $Y$, the map is said to be an homomorphism if for any $x_1,x_2\in X$, $T(x_1x_2) = T(x_1)T(x_2)$.
Representation. A representation $T$ of a group $G$ is an homomorphism from $G$ to $GL(V)$.
Invariant subspace. If $T$ is a representation of the group $G$ in the vector space $V$, $U\subset V$ is said to be an invariant subspace if $\forall g\in G,\forall v\in V, T(g)v\in U$.
Orthogonal subspace. Let $V$ be the vector space associated with the unitary representation $T$ and $W$ be the invariant subspace of $V$ through $T$. One defines the orthogonal subspace $W^\bot$ as {y : $x^\bot y = 0$, $\forall x\in W$}.
In my course, the prof shows the following property.
Property shown. Let V be the vector space associated to the unitary representation $T$ and let $W$ be a subspace of $V$. If $W$ is an invariant subspace of $V$ through $T$ then the associated orthogonal subspace (denoted $W^\bot$) is invariant as well.
I am pretty sure that one could make this a more general statement, by inserting an if and only if condition. I think I showed the property from the other side, but I would like for someone to review my proof.
Property I want to show. Let $V$ be the vector space associated with the unitary representation $T$ and let $W$ be a subspace of $V$. If $W^\bot$ is an invariant subspace of $V$ through $T$, then $W$ is invariant as well.
Proof. Let $x\in W$ and $y\in W^\bot$.
\begin{equation}
\forall g\in G, (T(g)x)^\dagger y = x^\dagger T^\dagger (g)y = x^\dagger T^{-1}(g)y
\end{equation}
The last equality holds true for any unitary operator.
\begin{equation}
x^\dagger T^{-1}(g)y = x^\dagger T(g^{-1})y = 0
\end{equation}
Where we used the fact that $W^\bot$ is an invariant subspace. We therefore have that $(T(x)x)^\dagger y = 0$, which means that $T(g)x\in W$ : we have shown that $W$ is an invariant subspace of $V$ through $T(g)$.
Is this proof correct? Moreover, can we, therefore, say that
Theorem. Let V be the vector space associated to the unitary representation $T$ and let $W$ be a subspace of $V$. Let $W^\bot$ be the associated orthogonal subspace. $W$ is invariant if and only if $W^\bot$ is invariant.
Best Answer
Regardless of the language problem, there are several points in your question that I think is mathematically unclear. You have basically omitted all assumptions so that many things have to be guessed.
Homomorphism. What you want to define is a homomorphism between groups, thus $X, Y$ are groups and it doesn't make sense to say that $T$ is a "linear map".
Invariant subspace. The correct definition is $\forall g\in G,\forall v\in U,T(g)v\in U$.
Orthogonal subspace. When talking about "unitary representation", you are assuming implicitly that the vector space $V$ is equipped with an inner product $\langle \cdot, \cdot \rangle$. Assuming that $V$ is a $\Bbb C$-vector space (which is usually the case but not always), one usually needs to assume that
If these assumptions are indeed assumed, then the result you state is true, as it is a simple consequence of duality, namely $(W^\bot)^\bot = W$ and hence you can apply what your prof showed to $W^\bot$.