To show that
$$
\left(\sigma\cdot\mathbf{n}\right)^2=\mathbf n\cdot\mathbf n+i\sigma\cdot\left(\mathbf n\times\mathbf n\right)\tag{1}
$$
consider writing the above as
\begin{align}
\left(\sigma\cdot\mathbf a\right)\left(\sigma\cdot\mathbf b\right)&=\sum_j\sigma_ja_j\sum_k\sigma_kb_k\\
&=\sum_j\sum_k\left(\frac12\{\sigma_j,\,\sigma_k\}+\frac12[\sigma_j,\,\sigma_k]\right)a_jb_k\\
&=\sum_j\sum_k\left(\delta_{jk}+i\epsilon_{jkl}\sigma_l\right)a_jb_k\tag{2}
\end{align}
where the 2nd line arises from using the anti-commutating and commutating relation for the matrices. In the third line, we have the Kronecker delta and Levi-Civita symbol. The result (1) follows from completing the math from (2) (that is, writing it in vector notation and replacing $\mathbf a$ and $\mathbf b$ with $\mathbf n$).
The remainder is to show that this is equal to 1. For that, the following two hints should suffice:
- Note that for two vectors $\mathbf a$ and $\mathbf b$, $\mathbf a\times\mathbf b=-\mathbf b\times\mathbf a$. What requirement is needed if $\mathbf b=\mathbf a$: $\mathbf a\times\mathbf a=?$
- For the unit vector, e.g. $\mathbf n=(1,\,0)^T$, what is the dot product?
I'll try to help here, with some details I believe to be of aid in this question. If there's anything in the need of fixing here, please let me know.
First of all, let's introduce some terminology. Given a group $G$ and a vector space $V$, a representation of $G$ on the vector space $V$ is a homomorphism $\rho : G\to GL(V)$. We then say that $V$ carries a representation of $G$.
If $V$ is a complex inner product space, and $\rho(g)$ is unitary for each $g\in G$, then $\rho$ is said a unitary representation.
It turns out the meaning of this is that the elements of $G$ act on the vector space $V$.
There are lots of constructions that can be acrried out with representations. One of them is the so-called direct sum of representations, which as can be expected is tied to the direct sum of vector spaces.
The point is that considering the group $G$ fixed if you have a representation $(\rho,V)$ and another one $(\sigma,W)$, then you can induce a representation $(\rho\oplus \sigma, V\oplus W)$ on the vector space $V\oplus W$ by setting
$$\rho\oplus\sigma(g)(v,w)=(\rho(g)v, \sigma(g)w).$$
It turns out direct sums allows you to decompose representations into smaller components that build up the representation you started with.
For that matter one defines a subrepresentation of $(\rho,V)$ to be given by a subspace $W$ of $V$ invariant under the $\rho$-action of $G$. It turns out then that $(\rho,W)$ is a representation itself. With this terminlogy we say that a representation is irreducible if it doesn't have any proper subrepresentations, i.e., if it doesn't have "smaller parts".
Now, let's talk about rotations. The rotation group in three-dimensions is $SO(3)$. Quantum Mechanics describes systems by Hilbert spaces, which are complex complete inner product spaces. Transformations should be implemented by unitary transformations, to keep the physical quantities intact.
Thus if we want to know how rotations affect the states of the systems we are studying, we need to consider the action of $SO(3)$ on the Hilbert space $\mathcal{H}$, requiring unitarity, that is, we must consider unitary representations of $SO(3)$.
This is what your $\mathcal{D}(R)$ is. The issue is that you want the irreducible representations of $SO(3)$. This can be tackled by studying its Lie Algebra, which is the angular momentum algebra. The result one finds is: (i) there are no infinite-dimensional unitary irreducible representations of $SO(3)$, and (ii) there is one finite-dimensional representation of $SO(3)$ for each integer $j$ with dimensionality $2j+1$. These representations are denoted $D^{j}$.
But these are just the irreducibles. You can build up representations with the direct sum. When you take the direct sum of two such representations, the new one will be given by a block-diagonal matrix.
So when you let $j$ run through all integer values for example, you are actually considering the representation given by taking the direct sum over all these values. When you fix one $j$, you are considering just one particular irreducible representation acting on a subspace of state space.
For more informations, search for the irreducible representations of $SO(3)$, and you'll probably find the whole construction rigorously made.
Best Answer
There is a general expression in my article A Compact Formula for Rotations as Spin Matrix Polynomials, SIGMA 10 (2014), 084, to the effect that, e.g., for the doublet, \begin{gather*} e^{i(\theta/2)(\hat{\boldsymbol{n}}\cdot\boldsymbol{\sigma})}=I_{2}\cos{\theta/2}+i(\hat{\boldsymbol{n}}\cdot\boldsymbol{\sigma})\sin{\theta/2}, \end{gather*} and the triplet, $j=1$, so $J_{3}=\mathrm{diag}(1,0,-1)$, \begin{gather*} e^{i\theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=I_{3}+i(\boldsymbol{\hat {n}}\cdot\boldsymbol{J})\sin{\theta}+(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})^{2}(\cos\theta-1) \\ \phantom{e^{i\theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}} =I_{3}+(2i\hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin(\theta/2))\cos (\theta/2)+\tfrac{1}{2}(2i\hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2))^{2}. \end{gather*}
For the quartet, $j=3/2$, \begin{gather} e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_4 \cos (\theta/2)\left(1+\tfrac{1}{2}\sin^2 (\theta/2)\right)+(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))\left(1+\tfrac{1}{6} \sin^2 (\theta/2) \right) \nonumber \\ \phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{} +\frac{1}{2!} \bigl (2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2) \bigr)^2 \cos (\theta/2)+\frac {1}{3!} \bigl (2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J}\sin (\theta/2) \bigr)^3. \label{quartet} \end{gather}
For the quintet, $j=2$, \begin{gather*} e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_5+(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2)) \cos(\theta/2)\left(1+\tfrac{2}{3}\sin^2(\theta/2)\right) \\ \phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{} +\frac{1}{2!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^2}\left(1+\tfrac{1}{3} \sin^2 (\theta/2)\right) \\ \phantom{e^{i \theta (\hat{\boldsymbol{n}}\cdot\boldsymbol{J})}=}{} +\frac{1}{3!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^3} \cos(\theta /2) +\frac{1}{4!} (2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^4. \end{gather*}
For the sextet, $j=5/2$, \begin{gather*} e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} = I_6 \cos(\theta/2)\left(1+ \tfrac{1}{2} \sin^2 (\theta/2+\tfrac{3}{8} \sin^4 (\theta/2)\right) \\ \phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{} +(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta /2))\left(1+\tfrac{1}{6}\sin^2(\theta/2) +\tfrac{3}{40}\sin^4(\theta /2)\right) \\ \phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{} +\frac{1}{2!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^2} \cos(\theta /2) \left(1+\tfrac{5}{6}\sin^2(\theta/2)\right) \\ \phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{} +\frac{1}{3!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^3}\left(1+\tfrac{1}{2}\sin^2(\theta/2) \right) \\ \phantom{e^{i \theta(\hat{\boldsymbol{n}}\cdot\boldsymbol{J})} =}{} +\frac{1}{4!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin(\theta/2))^4}\cos(\theta /2) +\frac{1}{5!} {(2i \hat{\boldsymbol{n}}\cdot\boldsymbol{J} \sin (\theta/2))^5}. \end{gather*}
etc...
There is a simple pattern and compact formula for arbitrary spin detailed in that paper.