$\mathrm{SO}(3)$ is a wonderful group, but it quickly got bored rotating vectors in space. It wanted to be used to help solve differential equations, and so desperately wanted to act on functions. Then one day, it realized it could act on functions by acting on their input.
For instance, it contains a 90 degree clockwise rotation in the $x$-$y$ plane. It takes $(x,y,z)$ and replaces it with $(y,-x,z)$.
Well, the function $2x+3y+7z$ can also be rotated! We just replace $x$ by $y$, $y$ by $-x$, and leave $z$ alone: $2x+3y+7z$ is rotated into $2y+3(-x)+7z = -3x + 2y + 7z$. If we consider the action on all these linear $Ax+By+Cz$ it becomes clear that $\mathrm{SO}(3)$ acts by 3×3 matrices on the vector space with basis $\{x,y,z\}$.
Yay, $\mathrm{SO}(3)$ can act on functions now, and it acts on those linear functions as 3×3 matrices.
What about quadratics? Well we could have $2x^2 + 3xy + 7y^2$. That same 90 degree rotation takes it to $2(y)(y) + 3(y)(-x) + 7(-x)(-x) = 7x^2 - 3xy + 2y^2$. Yay, now $\mathrm{SO}(3)$ acts as 6×6 matrices on the quadratic polynomials $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$ spanned by $\{ x^2, xy, y^2, xz, yz, z^2 \}$.
It turns out though that $\mathrm{SO}(3)$ doesn't swirl these functions around very thoroughly. If you take a function, and rotate its input, all it does is rotate the laplacian. In particular, notice that it takes any harmonic polynomial (one with 0 laplacian) to a harmonic polynomial. For quadratic polynomials, this is particularly easy to describe:
When you use $\mathrm{SO}(3)$ to rotate $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$, it leaves the laplacian, $A+C+F$, alone. So it will never rotate $x^2$ into $xy$. In fact, it leaves the 5-dimensional space spanned by $\{ x^2-y^2, y^2-z^2, xy, xz, yz \}$ invariant. If we write things in these coordinates, then we get $\mathrm{SO}(3)$ acting as 5×5 matrices. In fact it acts irreducibly.
What happened to the 6th dimension? Well, it is spanned by a very silly function, the sphere: $x^2+y^2+z^2$. If you rotate the sphere, you get the sphere. On this one dimensional space, $\mathrm{SO}(3)$ is represented by 1×1 matrices. Well, actually, matrix. Every single rotation in $\mathrm{SO}(3)$ acts as the identity matrix $[1]$, since every rotation leaves the sphere alone.
If we use colors, like pink, lime, and periwinkle to describe vectors, then a polynomial like $5x^2-5y^2 + 6x + 3$ is composed of a pink term, $5x^2-5y^2$, where $\mathrm{SO}(3)$ acts in a 5×5 manner, a lime term, $6x$, where $\mathrm{SO}(3)$ acts in a 3×3 manner, and a periwinkle term, $3$, where $\mathrm{SO}(3)$ acts in a 1×1 manner.
A polynomial like $7x^2+5y^2$ is a little trickier to see its colors (it is a good thing $\mathrm{SO}(3)$ is so clever), $7x^2+5y^2 = 4(x^2+y^2+z^2) + 3(x^2-y^2) + 4(y^2-z^2)$. The fist term $4(x^2+y^2+z^2)$ is periwinkle where $\mathrm{SO}(3)$ acts in a 1×1 manner, but the next two terms $3(x^2-y^2) + 4*(y^2-z^2)$ are pink where $\mathrm{SO}(3)$ acts in a 5×5 manner.
To clarify what I am specifically interested in knowing, if I have found some irreducible representations of a group $G$. Say I have $\chi_1, \dots, \chi_m$. I know I haven't found all of them because I know the number of conjugacy classes. Then, say, I some other non-irreducible character $\chi$ and I know, say, that this is the character of some representation. Then I subtract a linear combination of $\chi_1, \dots, \chi_m$, and define the class function $\psi = \chi - (a_1\chi_1 + \dots + a_m\chi_m)$. How do I know whether this $\psi$ is the character of some representation?
This question is much easier than your general question; assuming the $a_i$ are nonnegative integers, the answer is if and only if $\langle \chi, \chi_i \rangle \ge a_i$ for all $i$. This follows from:
Lemma: A class function $\chi$ is the character of a representation iff for every irreducible character $\chi_i$, $\langle \chi, \chi_i \rangle$ is a nonnegative integer.
Proof. If $\chi$ is the character of a representation $V$ then $\langle \chi, \chi_i \rangle$ is the multiplicity of the irreducible representation $V_i$ corresponding to $\chi_i$ in $V$, so this condition is clearly necessary. On the other hand, if this condition holds, then $\chi = \sum \langle \chi, \chi_i \rangle \chi_i$, and hence the direct sum of $\langle \chi, \chi_i \rangle$ copies of $V_i$ is a representation with character $\chi$. $\Box$
If you set $a_i = \langle \chi, \chi_i \rangle$ then you've removed the components of the representation corresponding to $\chi$ which correspond to the irreducibles with character $\chi_i$. So all you're left with is the components corresponding to the irreducibles you haven't found yet.
Best Answer
The norm of the character is given by $$\frac{1}{|G|}\sum_g \chi(g)\chi^*(g)$$ Here $|G|=16$ and the sum is $4+4+4+4$, so the norm is actually $1$, not $2$. So there isn't a contradiction as that 2D representation (I assume you are thinking of the standard representation in terms of Pauli matrices) is irreducible.