I want to find the $R_x \in \mathbb{R}^{3 \times 3}$, that minimizes
$$
||(R_1R_x-R_xR_2)||_F^2
$$
subject to the orthonormality constraint
$$
R_x^T R_x=I
$$
where $R_1$, $R_2$ and $R_x$ are all rotation matrixes (orthogonal and $det(\cdot)=1$), $R_1$ and $R_2$ are both known.
I tried:
I apply the Lagrange multiplier method to the problem:
$$
min \quad J(R_x) = {\rm tr}((R_1R_x-R_xR_2)^T(R_1R_x-R_xR_2)+(R_x^TR_x-I)Y)
$$
where $Y$ is a symmetric matrix.
I take the derivative of the equation above and set it equal to zero:
$$
\frac{\partial(J(R_x))}{\partial(R_x)} = R_x(4I+2Y)-2R_1^TR_xR_2-2R_1R_xR_2^T=0 \\
R_1^TR_xR_2+R_1R_xR_2^T =R_x(2I+Y)
$$
so I'm stuck here. I don't know how to solve a matrix equation like this, involving an unknown symmetric matrix $Y$.
Can anyone help me or give me some suggestions?
Thanks very much in advance.
New:
I found a Question that is almost the same as mine solving-for-a-rotation-matrix-equation.
According to that answer, maybe I was wrong to use Lagrange multiplier method.
And solving the matrix equation:
$$R_1R_x=R_xR_2$$
where $R_1$, $R_2$ and $R_x$ are all rotation matrixes.
@Ben Grossmann's solution is using spectral decomposition or Schur decomposition such as QR.
I'm sorry for my poor knowledge of matrix theory. I still don't know the exact solution.
Can anyone tell me the more clear solution?
Any help would be greatly appreciated.
Best Answer
To expand on my earlier answer, here is a solution that works in the case that an exact solution is possible (i.e. the minimum is zero). Unlike the solution that I had in mind at the time, this does not involve the use of complex numbers.
As I state in my post, a rotation $R_x$ such that $R_1R_x = R_xR_2$ exists if and only if $R_1,R_2$ are rotations by the same angle, possibly about different axes. Let $u_1$ denote an axis of rotation of $R_1$ (one of the two unit vectors from the null space of $R_1 - I$). Extend this vector to a right-handed orthonormal basis $(u_1,u_2,u_3)$ of $\Bbb R^3$. Let $U$ denote the matrix whose columns are $u_1,u_2,u_3$. We must have $$ U^TR_1U = \pmatrix{1 & 0 & 0\\0 & \cos \theta & -\sin \theta\\ 0 & \sin \theta & \cos \theta} $$ for some angle $\theta \in [0,2 \pi)$ (incidentally, we must have $\operatorname{tr}(R_1) = 1 + 2 \cos \theta$).
Let $v_1$ denote an axis of rotation of $R_2$. Extend this vector to a right-handed orthonormal basis $(v_1,v_2,v_3)$ of $\Bbb R^3$. Let $V$ denote the matrix whose columns are $v_1,v_2,v_3$. One of two equations must hold: $$ V^TR_2V = \pmatrix{1 & 0 & 0\\0 & \cos \theta & \sin \theta\\0 & - \sin \theta & \cos \theta} \quad \text{or} \quad V^TR_2V = \pmatrix{1 & 0 & 0\\0 & \cos \theta & -\sin \theta\\0 & \sin \theta & \cos \theta}. $$ If the first equation holds, replace $v_1$ with $-v_1$ and $v_2$ with $-v_2$. With the new matrix $V$, the second equation now holds.
With that, we have found rotation matrices $U$ and $V$ such that $$ U^TR_1U = V^TR_2V \implies R_1 UV^T = UV^T R_2. $$ Thus, we can take $R_x = UV^T$.
A bit of detail about $U$ and $V$:
One approach is as follows. Write $u_1 = (u_{11},u_{21},u_{31})$. We want any vector $u_2$ that is perpendicular to $u_1$; if $u_1$ is not $\pm (0,0,1)$, then one such option is to take $v = (u_{21},-u_{11},0)$ and let $u_2$ be the normalized vector in this direction, $u_2 = v/\|v\|$. To get the $u_3$ that makes for a right-handed orthonormal basis, we can compute the cross-product $$ u_3 = u_1 \times u_2. $$