[Math] Invariant Subspace and orthogonal complement

linear algebra

I found the following question:

Let $V$ be a finite dimensional inner product space over $C$ and let $T$ in $A(V)$ be unitary transformation. Prove that if $U$ is a subset of $V$ and it is $T$-Invariant then also the orthogonal complement of $U$ is $T$-Invariant.

What does the set $A(V)$ means?

I tried to do the following:
Let $x$ be in the orthogonal complement of $U$, we need to show that for every y in $U$
$(T(x),y) = 0$.
If $T = T^*$ then $(T(x),y) = (x,T^*(y)) = (x,T(y)) = 0$
but in this case $T^* = T-1$ so $(T(x),y) = (x,T-1(y))$ but I don't know how to continue from here.

Best Answer

No idea what $A(V)$ means, but if $U$ is $T$-invariant (i.e. $T(u) \in U$ for all $u \in U$), then $U^{\top} = \{ w \in V \, | \, \forall u \in U, \quad (w,u) = 0 \}$ will also be $T$-invariant.

Since $T$ is unitary, there exists $S$ such that $T \circ S$ is the identity map on $V$, so that $T$ is an isomorphism of $V$ to itself. By $T$-invariance, the restriction of $T$ to $U$ is an isomorphism of $U$ to itself, because $T$ maps $U$ to $U$ and is injective, but being a linear map of finite-dimensional vector spaces it must also be surjective.

This being said, if you take $w \in U^{\top}$ and an arbitrary element of $U$, write it in the form $u = T(u')$, then $T(w)$ is also in $U^{\top}$ because $(T(w), T(u')) = (w,u') = 0$. Since $u = T(u')$ was arbitrary, we conclude $T(w) \in U^{\top}$ and $U^{\top}$ is $T$-invariant.

Hope that helps,