I have a professor who says that I should be able to show a representation is irreducible simply by looking at its trace (with other possible conditions), but after researching this for a while, I have still not been able to see this.
I found a book in the library that considered the following representation of degree 2 given by
$\hspace{90pt} U(x) = \left( \begin{array}{ccc}
0 & -1 \\
1 & 0 \end{array} \right) \hspace{10pt}$ and $\hspace{10pt}U(y) = \left( \begin{array}{ccc}
0 & -1 \\
1 & 0 \end{array} \right)$
then showed that there were complex numbers $\lambda, \mu, \alpha, \beta$ such that
$\hspace{71pt} AU(x)A^{-1} = \left( \begin{array}{ccc}
\lambda & 0 \\
0 & \mu \end{array} \right) \hspace{10pt}$ and $\hspace{10pt} AU(y)A^{-1} = \left( \begin{array}{ccc}
\alpha & 0 \\
0 & \beta \end{array} \right) \hspace{10pt}$
where $A=\left( \begin{array}{ccc}
a & b \\
c & d \end{array} \right)$ with determinant not $0$.
However, by taking determinants of both sides for each, and then taking the trace of each as well, $\mp i a=\pm a$. This only occurs when $a=0$ which implies $c=0$ making the $\det(A)=0$. This was a contradiction, proving that $U$ was irreducible.
Does this kind of thing work in general? The book also talked about character tables, but I'm not really sure how they relate to the reducibility of a representation. I'm very new at this and my only background has been from Wikipedia, various webpages, and some books from the library. Any help would be appreciated.
The book is "Groups, Representations, and Characters" by Victor E. Hill.
Best Answer
Let $G$ be a group with $n$ elements, and let $\chi_R(g)$ be the trace of the matrix for $g$ in the representation $R$. So we have a vector for $R$: $$(\chi_R(g_1),\ldots,\chi_R(g_n))$$ which I'll denote just $\chi_R$. Now define an inner product on these vectors: for any representations $R$ and $S$, $$\langle \chi_R,\chi_S\rangle = \frac{1}{n}\sum_{g\in G}\chi_R(g)\overline{\chi_S(g)}$$ One of the big results about characters is that for any two inequivalent irreducible representations, $\langle \chi_R,\chi_S\rangle = 0$, and for any irreducible $R$, $\langle \chi_R,\chi_R\rangle = 1$. This is known as the orthogonality of irreducible characters.
Now suppose a representation $T$ is reducible. To keep notation simple, let's do the case where $T$ is the direct sum of two irreducible representations $R$ and $S$ (possibly equivalent). It's not hard to see that then $\chi_T = \chi_R+\chi_S$. By linearity of the inner product and the orthogonality just mentioned, if $R$ and $S$ are inequivalent $$\langle \chi_T,\chi_T\rangle = \langle \chi_R+\chi_S,\chi_R+\chi_S\rangle = 1+1 = 2$$ On the other hand, if $R$ and $S$ are equivalent, then $\chi_R = \chi_S$ (another big result about characters), so $\chi_T=2\chi_R$ and $$\langle \chi_T,\chi_T\rangle = 4\langle \chi_R,\chi_R\rangle = 4$$ The same argument works if $T$ is a sum of more than two irreducible characters, the notation just gets a bit messier.
So irreducible characters are characterized by $\langle \chi_R,\chi_R\rangle = 1$.
Here are a couple of examples. First, the simplest reducible case: the trivial 2-dim representation of the group with one element, $G=\{1\}$. If $V$ is the vector space, with basis (say) $\{v_1,v_2\}$, then $V$ is the direct sum $[v_1]\oplus[v_2]$ (I'm using [] for "the span of", so $V=[v_1,v_2]$). Now $[v_1]$ and $[v_2]$ are obviously invariant subspaces, so we have an invariant direct-sum decompositon, which makes the representation reducible. This reducibility shows up in the matrix for the sole element of $G$, which is $$\left[\begin{align*}\fbox{1} & 0 \\ 0 & \fbox{1}\end{align*}\right]$$ The boxes are the matrices in the 1-dim representation of $G$. Notice the block form: $$\left[\begin{align*}\fbox{$M_{11}$} & 0 \\ 0 & \fbox{$M_{22}$}\end{align*}\right]$$ Writing $T$ for the 2-dim rep, $\langle \chi_T,\chi_T\rangle = 4$. Notice that this a special case of the $\chi_T=2\chi_R$ situation described above.
Second example: let $G=S_3$. The 2-dim rep for $G$ comes from treating $G$ as the symmetry group of an equilateral triangle: each of the symmetries can be realized by an invertible linear transformation of the plane. (They also happen to be isometries.) Let's say the elements of $G$ are $\{1,r,r^2,f_1,f_2,f_3\}$ where $r$ is a $120^\circ$ rotation and $f_1,f_2,f_3$ are the three "flips" (reflexions). The vector $\chi_T$ looks like this: $(2, -1, -1, 0, 0, 0)$. (I leave it as an exercise to show that these entries are correct.) Now we compute: $$\langle \chi_T,\chi_T\rangle = \frac{1}{6}\left(2^2+(-1)^2+(-1)^2+0^2+0^2+0^2\right) = 1$$ So the representation $T$ is irreducible, which is pretty easy to see directly.
I'm not familiar with Hill's book, but I would imagine it has the results about characters I mentioned. There is a set of on-line notes I'd recommend: Finite Groups .