If we have three points like $A(x_1,y_1,z_1)$, $B(x_2,y_2,z_2)$ and $C(a,b,c)$. Then, $A$ and $B$ determines a line like $l$. After that, we rotate $C$ around $l$ by $\omega$ degree (anti-clockwise). How can be calculated new position of $C$ "$C'(a',b',c')$"?
[Math] Rotation around a line which is determined by two points in 3D space
analytic geometrygeometry
Related Solutions
[Edit: the following answer addresses the case of the line through the center of the sphere.]
Note: there is an ambiguity due to the fact that "angle $\theta$" does not specify which way the rotation is going (you can call the North Pole the South Pole, and suddenly the Earth rotate in the other way!)
What I will assume is that you are given the center $(a,b,c)$ of the sphere and a vector $(\alpha,\beta,\gamma) \neq (0,0,0)$ so that $x_2 = a + \alpha$, $y_2 = b + \beta$, and $z_2 = c + \gamma$. We shall assume the rotation is right handed relative to the vector $(\alpha,\beta,\gamma)$, that is, if you look along the direction given by $(\alpha,\beta,\gamma)$, the rotation is clockwise. Note that for any $\lambda \neq 0$, $(\alpha,\beta,\gamma)$ and $(\lambda \alpha,\lambda\beta,\lambda\gamma)$ determine the same line. But if $\lambda < 0$ their rotational directions are opposite.
For convenience we will require that the vector $(\alpha,\beta,\gamma)$ is a unit vector, that is $\alpha^2 + \beta^2 + \gamma^2 = 1$. We can always get this by dividing the vector by its length.
Then we can use this formula here plus a translation: a point $(x,y,z)$ is sent to $$ \begin{pmatrix} x \\ y \\ z\end{pmatrix} \mapsto \begin{pmatrix} a \\ b \\ c\end{pmatrix} + \begin{pmatrix} \cos \theta + (1 - \cos \theta)\alpha^2 & \alpha\beta(1-\cos\theta) - \gamma \sin\theta & \alpha\gamma(1-\cos\theta) + \beta \sin\theta \\ \alpha \beta(1-\cos\theta) + \gamma \sin\theta & \cos\theta + (1-\cos\theta)\beta^2 & \beta\gamma(1-\cos\theta) - \alpha\sin\theta\\ \alpha\gamma(1-\cos\theta) - \beta \sin\theta & \beta\gamma(1-\cos\theta) + \alpha\sin\theta & \cos\theta + (1-\cos\theta) \gamma^2 \end{pmatrix} \begin{pmatrix} x - a \\ y - b \\ z - c\end{pmatrix} $$
Let $r_1=PQ_1$ and $r_2=PQ_2$. You have then two equations for the unknowns $r_1$, $r_2$: $$ r_1\tan\beta1-r_2\tan\beta_2=z_1-z_2; \quad r_1^2+r_2^2-2r_1r_2\cos\gamma=(x_1-x_2)^2+(y_1-y_2)^2. $$ You can solve and find $r_1$, $r_2$. You can then obtain the coordinates of $P$ from: $$ (x_p-x_1)^2+(y_p-y_1)^2=r_1^2; \quad (x_p-x_2)^2+(y_p-y_2)^2=r_2^2; \quad z_p=z_1-r_1\tan\beta_1. $$
Best Answer
There are several ways to do it, it depends on what you want to do in practice. With linear algebra, this can be achieved using a simple matrix multiplication. You can define a unit vector from $A$ to $B$ to define a rotation axis, then apply the axis-angle rotation matrix on the vector of the point $C$.
This can also be done using quaternions.
See axis-angle representation for more information.