The matrix elements for the $(2l+1)$-dimensional irreducible representation of SO(3) are given by:
$D_{m',m}^l(\phi_1,\Phi,\phi_2)=[i^{m'-m}\sqrt{(l+m')!(l-m')!(l+m)!(l-m)!}$
$\times(\frac{1+\cos\Phi}{2})^l (\frac{1-\cos\Phi}{1+\cos\Phi})^{(m'-m)/2}$
$\times\sum\limits_k \frac{(-1)^k}{(l+m-k)!(l-m'-k)!(m'-m+k)!k!}$
$\times(\frac{1-\cos\Phi}{1+\cos\Phi})^k] e^{im'\phi_2}e^{im\phi_1}$
My question is, how can there be multiple irreducible representations of SO(3) each of which has a different dimension? I thought that by definition an irreducible representation is one which has no non-trivial invariant subspace. I know that SO(3) can be represented by the group of 3×3 symmetric matrices of determinant = 1 (I thought this had something to do with the fact that it takes 3 parameters to describe a 3D rotation). If I have a larger matrix (e.g. 5×5) doesn't it have to have a non-trivial invariant subspace (e.g. of dimension 2), and therefore be reducible?
The application here is the expansion of functions using generalized spherical harmonics, which is accomplished (per the Peter-Weyl theorem) using an infinite linear combination of the matrix elements of the irreducible representationS (emphasis on the S) of SO(3).
Best Answer
$\mathrm{SO}(3)$ is a wonderful group, but it quickly got bored rotating vectors in space. It wanted to be used to help solve differential equations, and so desperately wanted to act on functions. Then one day, it realized it could act on functions by acting on their input.
For instance, it contains a 90 degree clockwise rotation in the $x$-$y$ plane. It takes $(x,y,z)$ and replaces it with $(y,-x,z)$.
Well, the function $2x+3y+7z$ can also be rotated! We just replace $x$ by $y$, $y$ by $-x$, and leave $z$ alone: $2x+3y+7z$ is rotated into $2y+3(-x)+7z = -3x + 2y + 7z$. If we consider the action on all these linear $Ax+By+Cz$ it becomes clear that $\mathrm{SO}(3)$ acts by 3×3 matrices on the vector space with basis $\{x,y,z\}$.
Yay, $\mathrm{SO}(3)$ can act on functions now, and it acts on those linear functions as 3×3 matrices.
What about quadratics? Well we could have $2x^2 + 3xy + 7y^2$. That same 90 degree rotation takes it to $2(y)(y) + 3(y)(-x) + 7(-x)(-x) = 7x^2 - 3xy + 2y^2$. Yay, now $\mathrm{SO}(3)$ acts as 6×6 matrices on the quadratic polynomials $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$ spanned by $\{ x^2, xy, y^2, xz, yz, z^2 \}$.
It turns out though that $\mathrm{SO}(3)$ doesn't swirl these functions around very thoroughly. If you take a function, and rotate its input, all it does is rotate the laplacian. In particular, notice that it takes any harmonic polynomial (one with 0 laplacian) to a harmonic polynomial. For quadratic polynomials, this is particularly easy to describe:
When you use $\mathrm{SO}(3)$ to rotate $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$, it leaves the laplacian, $A+C+F$, alone. So it will never rotate $x^2$ into $xy$. In fact, it leaves the 5-dimensional space spanned by $\{ x^2-y^2, y^2-z^2, xy, xz, yz \}$ invariant. If we write things in these coordinates, then we get $\mathrm{SO}(3)$ acting as 5×5 matrices. In fact it acts irreducibly.
What happened to the 6th dimension? Well, it is spanned by a very silly function, the sphere: $x^2+y^2+z^2$. If you rotate the sphere, you get the sphere. On this one dimensional space, $\mathrm{SO}(3)$ is represented by 1×1 matrices. Well, actually, matrix. Every single rotation in $\mathrm{SO}(3)$ acts as the identity matrix $[1]$, since every rotation leaves the sphere alone.
If we use colors, like pink, lime, and periwinkle to describe vectors, then a polynomial like $5x^2-5y^2 + 6x + 3$ is composed of a pink term, $5x^2-5y^2$, where $\mathrm{SO}(3)$ acts in a 5×5 manner, a lime term, $6x$, where $\mathrm{SO}(3)$ acts in a 3×3 manner, and a periwinkle term, $3$, where $\mathrm{SO}(3)$ acts in a 1×1 manner.
A polynomial like $7x^2+5y^2$ is a little trickier to see its colors (it is a good thing $\mathrm{SO}(3)$ is so clever), $7x^2+5y^2 = 4(x^2+y^2+z^2) + 3(x^2-y^2) + 4(y^2-z^2)$. The fist term $4(x^2+y^2+z^2)$ is periwinkle where $\mathrm{SO}(3)$ acts in a 1×1 manner, but the next two terms $3(x^2-y^2) + 4*(y^2-z^2)$ are pink where $\mathrm{SO}(3)$ acts in a 5×5 manner.