By following the answer from this question, for a given plane $P$ (defined with three points $A$, $B$ and $C$) I manged to rotate the objects from it into the plane "$z=0$" with the following method:
$$\mathbf w = {\mathbf n\over\|\mathbf n\|}, \mathbf u={B-A\over\|B-A\|}, \mathbf v=\mathbf w\times\mathbf u.$$
Where plane’s normal is $\mathbf n=(B-A)\times(C-A)$ and the matrix $R=\begin{bmatrix}\mathbf u&\mathbf v&\mathbf w\end{bmatrix}$ represents a rotation that maps the standard basis vectors onto plane "$z=0$" (for more details of the approach you can read the description in the posted link above).
I need to do the same thing for rotating the objects from plane $P$ to the planes "$x=0$" and "$y=0$", so in these cases how the rotation vector should look like?
Best Answer
I have found the solution which I was looking for. If the rotation vector is: $R=\begin{bmatrix}\mathbf u&\mathbf v&\mathbf w\end{bmatrix}$, for rotation to all three planes, here are the formulas for rotations:
to $XY$ plane:
$$\mathbf n=(B-A)\times(C-A),\mathbf w = {\mathbf n\over\|\mathbf n\|}, \mathbf u={B-A\over\|B-A\|}, \mathbf v=\mathbf w\times\mathbf u.$$
to $XZ$ plane:
$$\mathbf n=(B-C)\times(A-C),\mathbf v = {\mathbf n\over\|\mathbf n\|}, \mathbf u={B-C\over\|B-C\|}, \mathbf w=\mathbf v\times\mathbf u.$$
to$YZ$ plane:
$$\mathbf n=(A-B)\times(C-B),\mathbf u = {\mathbf n\over\|\mathbf n\|}, \mathbf w={A-B\over\|A-B\|}, \mathbf v=\mathbf u\times\mathbf w.$$