This is an old question that has many answers and approaches through the site. I'm trying to understand this: What are the irreducible representations of $S_3$ over $\mathbb{C}^3$?
I'm stuck in the part of proving that the two-dimensional representation spanned by the vectors $\lbrace(1,-1,0)^T, (0,1,-1)^T \rbrace$ is irreducible. If I understand correctly, one way to prove this is to give e.g. two representations of say permutations $\pi(\sigma_{12}), \pi(\sigma_{34})$ and see if there's a common basis that diagonalizes them simultaneously. If their eigenvectors have "different structure", then the representation is irreducible.
So the point is to compute the action of $\pi(\sigma_{12}), \pi(\sigma_{34})$ using the basis $\lbrace(1,-1,0)^T, (0,1,-1)^T \rbrace$. In particular, it is claimed that $\pi(\sigma_{12})$ has representation $\begin{pmatrix}-1 & 1\\0&1 \end{pmatrix}$. As far as I understand, one computes this representation by only using the standard inner product in $\mathbb{C}^3$ (the inner product is invariant under $\pi(\sigma)$ over $\mathbb{C}^3$ because each $\pi(\sigma)$ is unitary). In any case, I obtain for instance
$$
[\pi(\sigma_{12})]_{21} = (0,1,-1)\cdot\begin{pmatrix}-1\\1\\0 \end{pmatrix}=1,
$$
but it is claimed that this matrix element must be zero. Is there's something wrong in what I did/assumed?
Best Answer
We have a linear transformation $T = \pi(\sigma_{12})$ and we want to know how to write down the matrix $A$ when $T$ is represented in a bad (skewed) basis. If it was a nice basis, then dot products would very easily give us our answer.
In principle, even in a bad basis it is easy. This technique works even if you use a different basis for the space before the operation $T$ than for the space after (the domain and the range). I'll use $\vec{e}$ to mean "before" and $\vec{f}$ to mean "after", but in your problem (and I think all of Quantum Mechanics) we would use the same basis, so $\vec{e}_1 = \vec{f}_1 = \alpha_1 = (1,-1,0)^T$ and $\vec{e}_2 = \vec{f}_2 = \alpha_2 = (0,1,-1)^T$. Of course we would also have used a unitary basis in QM :-)
How to compute the matrix
Take each basis vector of the domain $\vec{e}_1$, $\vec{e}_2$, etc. and apply $T$ to get some answer. Then write that answer in terms of the basis vectors of the range, $\vec{f}_1$, $\vec{f}_2$, etc. The coefficients form the columns of $A$.
For example if $T(\vec{e}_1) = - \vec{f}_1$ then the first column (first coming from the "1" by the "e") is $\begin{bmatrix} -1 \\ 0 \end{bmatrix}$.
If $T(\vec{e}_2) = \vec{f}_1 + \vec{f}_2$, then the second column is $\begin{bmatrix} 1 \\ 1 \end{bmatrix}$.
So $A = \begin{bmatrix} \uparrow & \uparrow \\ T(\vec{e}_1) & T(\vec{e}_2) \\ \downarrow & \downarrow \end{bmatrix} = \begin{bmatrix} -1 & 1 \\ 0 & 1 \end{bmatrix}$ where I've pretended $T(\vec{e}_1)$ and $T(\vec{e}_2)$ are literally the coefficients instead of the combinations of basis vectors.
Why does it work?
The reason is mostly clear when you multiply the matrix times a coordinate vector. If we represent a vector $\vec{v} = a \vec{e}_1 + b \vec{e}_2$ in coordinates it is just $\begin{bmatrix} a \\ b \end{bmatrix}$. In particular, $\vec{e}_1$ is represented by $\begin{bmatrix} 1 \\ 0 \end{bmatrix}$ and if $A=\begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix}$ then $T(\vec{e}_1)$ is represented by $\begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix} \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} 1 A_{11} + 0 A_{12} \\ 1 A_{21} + 0 A_{22} \end{bmatrix} = \begin{bmatrix} A_{11} \\ A_{21} \end{bmatrix}$
In other words, $\vec{e}_1$ just takes the first column. A general version is nice to remember:
$\begin{bmatrix} A_{11} & A_{12} \\ A_{21} & A_{22} \end{bmatrix} \begin{bmatrix} a \\ b \end{bmatrix} = a \begin{bmatrix} A_{11} \\ A_{21} \end{bmatrix} + b \begin{bmatrix} A_{12} \\ A_{22} \end{bmatrix}$
The entries in the coordinate vector tell you how much of each column of $A$ to use for the answer.
Why do we bother with nice bases? (Why are bad bases bad?)
In this small example, I hope you think "that was not so hard". Why then do we call a skewed basis bad? The reason is that converting $T(\vec{e})$ into the coefficients of the basis elements $\vec{f}_i$ can be very hard!
In a unitary basis, you just use dot products to find coefficients. In a bad basis, you "solve a linear system", possibly using Gaussian elimination or some other technique. It is not super hard, but it is a lot harder than a dot product.