I am trying to do a problem from artin's algebra 2nd ed (Chapter 10, Exercise 2.3) but having trouble:
Let $(\rho , V)$ be a representation of the symmetric group $S_3$. Let $x=(123), y=(12)$ be the usual generators of $S_3$.
(a) Let $u\neq 0$ be a vector in $V$. Let $v=u+xu+x^2u$ and $w=u+yu$. By analyzing the $S_3$-orbits of $v,w$, show that $V$ contains a nonzero invariant subspace of dimension $\leq 2$.
(b) Prove that all 2-dimensional irreducible representations of $S_3$ are isomorphic, and determine them all.
After some calculations, i found that $xv=v$ and $yw=w$. But I dont know what to do next. Also, we are not allowed to use character theory, thanks in advance.
Best Answer
We have $$x^3 = y^2 = (xy)^2 = 1,$$ so $$yx = (yx)^{-1} = x^2y,\text{ }yx^2 = x^2yx = x^4y = xy.$$
a.
Fix a nonzero $u\in V$, and let $$v = u+xu+x^2u = (1+x+x^2)u,\text{ }w=u+yu = (1+y)u.$$ Since $\{1,x,x^2\},\{1,y\}$ are subgroups of $G = S_3$, we have $xv = x^2v = v$ and $yw = w$.
We have the operator identities $$y(1+x+x^2) = y+yx+yx^2 = y+x^2y+xy = (1+x+x^2)y,$$ $$x(1+y) = x+xy = x+yx^2,$$$$x^2(1+y) = x^2+x^2y = x^2+yx.$$ It follows that $yv = (1+x+x^2)(yu)$, so $x,x^2$ fix $yv$ $($just as they fix $v$$)$. Of course, $y(yv) = y^2v = v$, so $\text{span}(\text{Orb}(v)) = \text{span}(v,yv)$ is a $G$-invariant subspace of dimension at most $2$. This gives us a nonzero invariant subspace of dimension at most $2$ if $v\ne0$.
Otherwise, if $v=0$, we analyze the $G$-orbit of $w$ instead. Since $$xw = (x+yx^2)u, \text{ }x^2w = (x^2+yx)u,$$ we have $$y(xw) = (yx+x^2)u = x^2w,\text{ }y(x^2w) = (yx^2+x)u = xw.$$
But $1+y$ commutes with $1+x+x^2$, so $$w+xw+x^2w = (1+x+x^2)yu = y(1+x+x^2)u = yv = 0$$ $($from $v=0$$)$. Thus $$\text{span}(\text{Orb}(w)) = \text{span}(w,xw,x^2w) = \text{span}(w,xw)$$ is $G$-invariant with dimension at most $2$, and we are fine as long as $w\ne0$.
Thus we have reduced the problem to the case in which $$(1+x+x^2)u = (1+y)u = 0$$ for all $($nonzero$)$ vectors $u\in V$. In this case, we must have $$1+\rho_y = 1+\rho_x+(\rho_x)^2 = 0$$ identically $($as operators$)$, so in particular $\rho_y$ must fix all one-dimensional eigenspaces. Now set $u$ to be an arbitrary eigenvector of $\rho_x$; then $\rho_x$, $\rho_y$ both fix the nonzero subspace $\text{span}(u)$ of dimension $1$, and we are done.
We remark that a simpler approach altogether is just to start with an eigenvector of $\rho_x$.
b.
First we classify all $1$-dimensional $($irreducible$)$ representations of $G$ $($over some $1$-dimensional $V = (v)$, $v\ne0$$)$. Write $xv = \alpha v$ and $yv = \beta v$ $($for complex $\alpha$, $\beta$$)$; then $$x^3v = y^2v = (xy)^2v = v,$$ so $$\alpha^3 = \beta^2 = (\alpha\beta)^2 = 1,$$ which forces $$\alpha^3 = \alpha^2 = 1\implies \alpha = 1,$$ and $\beta = \pm1$. Thus the only $1$-dimensional $($irreducible$)$ representations of $G$ are the trivial $($$\beta=1$$)$ and sign $($$\beta=-1$$)$ representations. $($In terms of matrices, $R_x = [\alpha]$ and $R_y = [\beta]$ for both cases.$)$
Now consider an irreducible representation of $G$ over a $2$-dimensional $V$. Take an eigenvector $v\ne0$ of $x$ with eigenvalue $\lambda$, so $xv=\lambda v$ and $\lambda^3 = 1$. Define $w = yv$, so $$yw = y^2v = v.$$ Then $$xw = xyv = yx^2v = y\lambda^2 v = \lambda^2 w.$$
Of course, $w\ne v$, or else $(v)$ would be a $1$-dimensional $G$-invariant subspace of $V$. Similarly, $\lambda\ne1$, or else we would have $xv=v$, $xw=w$, $yv+yw = w+v$, and $(v+w)$ would be a $1$-dimensional $G$-invariant subspace.
Also, without loss of generality, we may assume $\lambda = e^{2\pi i/3}$, or else we could have started with the eigenvector $w\ne0$ of $x$ $($with eigenvalue $\lambda^2$$)$ instead of $v$. It follows that for any irreducible representation of $G$ over a $2$-dimensional $V$, there exist basis vectors $v$, $w\in V$ such that the matrices $R_x$, $R_y$ of $\rho_x$, $\rho_y$ with respect to the basis $B = (v,w)$ are $$\begin{pmatrix} e^{2\pi i/3} & 0 \\ 0 & e^{-2\pi i/3} \end{pmatrix},\, \begin{pmatrix} 0 & 1 \\ 1 & 0 \end{pmatrix},$$ respectively, so we are done.
We remark that it is easy to check that the three representations presented above are in fact irreducible. The one-dimensional representations are clearly irreducible; for the two-dimensional one, we use the matrix interpretation $R_x$, $R_y$ and note that a $G$-invariant one-dimensional subspace $(v)$ $($with $v = (p,q)^\text{T}$$)$ would have to have $(q,p) = yv\in (v)$ and thus $p^2 = q^2 \ne0$. But it is easy to check that $R_x$ sends $(p,p)$ to $(\lambda p, \lambda^{-1} p)\notin (v)$ and $(p,-p)$ to $(\lambda p,-\lambda^{-1} p)\notin (v)$.