Given a 3 by 3 matrix $M$ I would like to find the rotation matrix $R$ minimizing the Frobenius norm:
\begin{equation}
\|R-M\|_F
\end{equation}
Is there a closed form solution for $R$, or is it possible to express $R$ as the solution to a linear system? I would like to avoid gradient descent if possible.
Best Answer
Let $M=U\Sigma V$ be the singular value decomposition of $M$, then $R=UV$. If you want $R$ to be a proper rotation (i.e. $\det R=1$) and $UV$ is not, replace the singular vector $\mathbf{u}_3$ associated with the smallest singular value of $M$ with $-\mathbf{u}_3$ in the $U$ matrix. An appropriate reference for this answer is:
N. J. Higham. Matrix nearness problems and applications. In M. J. C. Gover and S. Barnett, editors, Applications of Matrix Theory, pages 1–27. Oxford University Press, 1989.