Find the matrix of the rotation in $\Bbb R^3$ through the angle $\alpha$ around the vector $(1, 2, 3)^T$ . We assume that the rotation is counterclockwise if we sit at the tip of the vector and looking at the origin.
Let $P(\vec x)$ be the linear transformation that rotates the coordinate system such that $z$ axis is in the direction of $(1, 2, 3)^T$, then by using some trignometry,
$P((0,0, 1)^T) = \dfrac1{\sqrt{13}}(1,2,3)^T$, $P((0,1, 0)^T) =(0,\sec \theta,\tan \theta)^T$ and $P((1,0,0)^T) =(\sec \theta,0,\tan \theta)^T$, where $\theta$ is the angle between $(1,2,3)^T$ and $(0,0,1)^T$.
Solving for $\theta$ and using matrix representation of $P$,
$$P(\vec x) = \begin{bmatrix}\dfrac{\sqrt{13}}{3} & 0 &\dfrac1{\sqrt{13}} \\ 0& \dfrac{\sqrt{13}}{3}& \dfrac2{\sqrt{13}} \\\dfrac{-2}{3} & \dfrac{-2}{3}& \dfrac3{\sqrt{13}} \end{bmatrix}\vec x$$
Therefore, I got the final rotation transformation as $T(\vec x) := (P^{-1}\circ R_\alpha \circ P)(\vec x)$, where $$R_\alpha(\vec x) = \begin{bmatrix}\cos \alpha & -\sin \alpha & 0 \\ \sin \alpha & \cos \alpha & 0 \\ 0&0&1\end{bmatrix}.$$
Basically what I did was to rotate the coordinate system with $P$ such that $z$-axis and $(1,2,3)^T$ coincide, then I rotated $xy$-plane by $\alpha$ and then reverted back to original coordinates.
Is this correct ?
Please don't use quaternions or any other voodoo like given here. Please use basic trignometry and linear algebra.
Sorry, normally I don't ask this type of question but I can't verify my results in this case.
Best Answer
You can find $R$ as $PSP^{-1}$ where $P$ is a rotation (orthonormal transformation) that maps the $x$-axis to the axial vector $(1,2,3)$, and $S$ is a rotation of angle $\alpha$ around the $x$-axis.
Finding $S$ is easy: $$S = \begin{bmatrix}1&0&0\\0&\cos\alpha&-\sin\alpha\\0&\sin\alpha&\cos\alpha\end{bmatrix}$$
Finding $P$ is more complex. Firstly, $Pe_1={(1,2,3)^T\over\sqrt{1^2+2^2+3^2}}$. Next, we meed to make sure that $Pe_i \perp Pe_j$ for all $i \neq j$, and $|Pe_1|=|Pe_2|=|Pe_3|=1$; for this we can use the cross product: Make $Pe_2={(1,2,3)^T\times e_1 \over |(1,2,3)^T \times e_1|}$, which is the unit vector perpendicular to $e_1$ and $(1,2,3)^T$, and similarly make $Pe_3 = {((1,2,3)^T\times e_1) \times (1,2,3)^T \over |(((1,2,3)^T\times e_1) \times (1,2,3)^T|}$, which is perpendicular to both $Pe_1$ and $Pe_2$ and of unit length. This satisfies all the conditions we need for $P$.
A remark: A key step in finding $P$ is finding a pair of vectors perpendicular to the axial vector and to each other. There is no continuous mapping that does this because of the hairy ball theorem. In spite of that, there are algorithms for doing this (see Yves Daoust's answer).