$\mathrm{SO}(3)$ is a wonderful group, but it quickly got bored rotating vectors in space. It wanted to be used to help solve differential equations, and so desperately wanted to act on functions. Then one day, it realized it could act on functions by acting on their input.
For instance, it contains a 90 degree clockwise rotation in the $x$-$y$ plane. It takes $(x,y,z)$ and replaces it with $(y,-x,z)$.
Well, the function $2x+3y+7z$ can also be rotated! We just replace $x$ by $y$, $y$ by $-x$, and leave $z$ alone: $2x+3y+7z$ is rotated into $2y+3(-x)+7z = -3x + 2y + 7z$. If we consider the action on all these linear $Ax+By+Cz$ it becomes clear that $\mathrm{SO}(3)$ acts by 3×3 matrices on the vector space with basis $\{x,y,z\}$.
Yay, $\mathrm{SO}(3)$ can act on functions now, and it acts on those linear functions as 3×3 matrices.
What about quadratics? Well we could have $2x^2 + 3xy + 7y^2$. That same 90 degree rotation takes it to $2(y)(y) + 3(y)(-x) + 7(-x)(-x) = 7x^2 - 3xy + 2y^2$. Yay, now $\mathrm{SO}(3)$ acts as 6×6 matrices on the quadratic polynomials $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$ spanned by $\{ x^2, xy, y^2, xz, yz, z^2 \}$.
It turns out though that $\mathrm{SO}(3)$ doesn't swirl these functions around very thoroughly. If you take a function, and rotate its input, all it does is rotate the laplacian. In particular, notice that it takes any harmonic polynomial (one with 0 laplacian) to a harmonic polynomial. For quadratic polynomials, this is particularly easy to describe:
When you use $\mathrm{SO}(3)$ to rotate $Ax^2+Bxy+Cy^2+Dxz+Eyz+Fz^2$, it leaves the laplacian, $A+C+F$, alone. So it will never rotate $x^2$ into $xy$. In fact, it leaves the 5-dimensional space spanned by $\{ x^2-y^2, y^2-z^2, xy, xz, yz \}$ invariant. If we write things in these coordinates, then we get $\mathrm{SO}(3)$ acting as 5×5 matrices. In fact it acts irreducibly.
What happened to the 6th dimension? Well, it is spanned by a very silly function, the sphere: $x^2+y^2+z^2$. If you rotate the sphere, you get the sphere. On this one dimensional space, $\mathrm{SO}(3)$ is represented by 1×1 matrices. Well, actually, matrix. Every single rotation in $\mathrm{SO}(3)$ acts as the identity matrix $[1]$, since every rotation leaves the sphere alone.
If we use colors, like pink, lime, and periwinkle to describe vectors, then a polynomial like $5x^2-5y^2 + 6x + 3$ is composed of a pink term, $5x^2-5y^2$, where $\mathrm{SO}(3)$ acts in a 5×5 manner, a lime term, $6x$, where $\mathrm{SO}(3)$ acts in a 3×3 manner, and a periwinkle term, $3$, where $\mathrm{SO}(3)$ acts in a 1×1 manner.
A polynomial like $7x^2+5y^2$ is a little trickier to see its colors (it is a good thing $\mathrm{SO}(3)$ is so clever), $7x^2+5y^2 = 4(x^2+y^2+z^2) + 3(x^2-y^2) + 4(y^2-z^2)$. The fist term $4(x^2+y^2+z^2)$ is periwinkle where $\mathrm{SO}(3)$ acts in a 1×1 manner, but the next two terms $3(x^2-y^2) + 4*(y^2-z^2)$ are pink where $\mathrm{SO}(3)$ acts in a 5×5 manner.
Best Answer
When we say that the 3-dimensional irreducible representation of $SO(3)$ is unique this is exactly what we mean. Precisely, it means that if you have two sets of 3-dimensional matrices that act as a group representation then you know automatically that those two sets are isomorphic. They may be distinct as sets but they are still identical as representations.
This applies even to vector spaces that are very different. Consider the following two vector spaces. The first is just $\mathbb{R}^3$. The second space is a subset of real functions on a sphere, spanned by the real spherical harmonics that have $\ell = 1$, $\mathcal{Y_1} = \operatorname{span}(Y_1^0(\theta,\phi),Y_1^{+1}(\theta,\phi),Y_1^{-1}(\theta,\phi))$. This is also a three-dimensional vector space, but clearly a very different set than the first. Both vector are representations of $SO(3)$ in the sense that there are linear operators on these spaces that represent the group. The orthogonal matrices you mention in your question are the operators for the first group. The operators that act on the second group are a little more abstract, but for every rotation $g$ that sends $(\theta,\phi)\mapsto (\theta',\phi')$ we can find a matrix much that $Y_1^m(\theta',\phi') = X^m_{\,\,m'}Y_1^{m'}(\theta,\phi)$
Despite the apparent difference between these two sets of matrices, $M$ and $X$, they are isomorphic as representations. That is, there is a map $T$ from $\mathbb{R^3}$ to $\mathcal{Y}_1$ that we can use to map between the two different sets of matrices: $M_g = T^{-1}X_g T$.
This is pretty abstract but it's important. It allows us to say that even though $\mathbb R^3 $ and $\mathcal{Y}_1$ are very different on some levels they are exactly the same when it comes to their behavior under rotations. Being clear about the difference between representations as sets and representations as abstract categories of behavior under rotations is important to getting a deeper understanding of representation theory.