In Anthony Zee's proof of Schur's lemma (in his book Group Theory in a Nutshell for Physicists, page 102), he used the following fact (summarized by myself) without proof:
Proposition: Let $G$ be a finite group, and $D(g)$ any unitary $n \times n$-matrix representation of $g \in G$. If $D$ is irreducible, then for any fixed $1 \le i,j \le n$, it cannot happen that $D_{ij}(g) = 0$ for all $g \in G$.
Although Zee demonstrated a reducible representation can have $D_{ij}(g) = 0$ for all $g \in G$, he did not show the converse. So I want to ask how to prove that? (Please try avoiding general theory of representations, which may involving modules, rings, etc; I am not familiar with those more abstract math…)
Best Answer
From now on $|i\rangle$ denotes the i-th element of the normalized canonical basis of the finite-dimensional Hilbert space $\mathbb{C}^n$ where the unitary nxn matrices $D(g)$ are defined. Furthermore, $$D_{ij}(g):= \langle i| D(g) |j\rangle.$$
Suppose that $D_{ij}(g) =0$ for all $g$ and a choice of $i,j$.
Notice that $i\neq j$ necessarily, otherwise we would have $|||i\rangle||=0 \neq 1$ when choosing $g=e$ (the unit element of the group).
The subspace $S$ of the finite linear combinations $$\sum_{k=1}^N c_k D(g_k)|i\rangle\quad N=1,2,\ldots\:, \quad g_k \in H\:, c_k \in \mathbb{C}$$ is invariant under the action of $D$ trivially, because $$D(f)D(g_k)|j\rangle =D(g'_k)|i\rangle := D(f\circ g_k)|j\rangle$$ extended to the whole S by linearity. Regarding the space $S$,
(1) it does not coincide with the whole vector space since it is made of vectors orthogonal to $|i\rangle$;
(2) it is not made of the zero vector only, as it includes the vector $|j\rangle \neq 0$.
Hence $S$ is an invariant proper subspace of $\mathbb{C}^n$. In other words $D$ is reducible. In summary,
PROPOSITION. If a unitary finite-dimensional representation $D$ on $\mathbb{C}^n$ ($n>1$) of a group $G$ is irreducible, then $D_{ij}(g) =0$ for all $g$ and a choice of $i,j$ cannot hold, where $$D_{ij}(g):= \langle i| D(g) |j\rangle$$ and $|i\rangle$ is the generic element of the canonical basis of $\mathbb{C}^n$.
The thesis and the proof is still valid if the space is infinite dimensional the representation $D$ is unitary by referring to a fixed Hilbert basis. Just replace the subspace above with its closure.
What about the converse statement? What it is possible to prove is the following fact.
PROPOSITION. Let $G \ni g \mapsto D(g)$ be a representation of the group $G$ made of unitary operators $D(g): H \to H$ on the Hilbert space $H$ ($\dim(H)>1$, also infinite dimensional and non-separable). If $D$ is reducible, then there is a Hilbert basis $\{u_j\}_{j\in J} \subset H$ such that $$\langle u_i| D(g) u_j\rangle =0 \quad \mbox{for all $g\in G$ and a pair $i,j \in J$ with $i\neq j$.}$$
PROOF. let $S \subset H$ be a (closed) proper invariant subspace, i.e. $D(g)(S) \subset S$ for every $g\in G$. Since the representation is unitary $D(g)(S^\perp) \subset S^\perp$ for every $g\in G$ and also $S^\perp$ is proper because $H= S\oplus S^\perp$ and $S \neq H$, $S\neq \{0\}$. Let $B$ be a Hilbert basis of $S$ and $B'$ a Hilbert basis of $S^\perp$. $B\cup B'$ is a Hilbert basis of $H$ by construction. Any pair of vectors one in $B$ and the other in $B'$ satisfies the thesis. QED