I have given a complex matrix $Q \in \mathbb{C}^{3,3}$ and I would like to find the "closest" rotation matrix (meaning $\mathcal{R}$ is orthogonal and $\det{(\mathcal{R})}=1$) $R \in \mathbb{R}^{3,3}$ which minimizes the norm
$$ \|Q-\mathcal{R}\|_F $$
Is there any closed form solution to this problem or an efficient algorithm I could use to compute $R$?
Background Information
I am adding some background information to this: In my application, I get a noisy sensor measurement $[B]_{meas}$:
$$ [B]_{meas}=\mathcal{R}[B]_{ref} + \mathcal{N}_B$$
$[B]_{ref}$ is a known, complex 3×3 matrix, $\mathcal{R}$ is an orthogonal, real-valued rotation matrix and $\mathcal{N}_B$ is an additive, complex valued noise matrix (3×3, complex). I am trying to get an estimate for $\mathcal{R}$ given the measurement $[B]_{meas}$.
Right now, the suboptimal method I use is the following:
$$ \operatorname*{argmin}_{\mathcal{R}} || \Re{([B]_{meas}[B]_{ref}^{-1}}) ||_{F}$$
I solve this optimization problem using the Procrustes method. However, I don't think I can use the Procrustes method directly because it will output an unitary matrix and not an orthogonal matrix.
Best Answer
Yes, there is.
Proof (Minimizer): Let $\langle \cdot,\cdot \rangle$ denote the Frobenius inner product. Write \begin{align} \|Q - W\|_F^2 &= \|Q\|_F^2 + \|W\|_F^2 - 2 \operatorname{Re}[\langle Q,W\rangle] \\ & = \|Q\|_F^2 + 3 - 2\operatorname{Re}[\langle A - Bi,W\rangle] \\ & = \|Q\|_F^2 + 3 - 2\operatorname{Re}[\langle A,W\rangle + i \langle B,W\rangle] \\ & = 3 + \|Q\|_F^2 - 2 \langle A,W \rangle. \end{align} Thus, minimizing the objective function is equivalent to maximizing $\langle A,W \rangle$ (over $W \in O(n)$). As is noted here, this maximum is attained with $W = UV^\top$.
Proof (Uniqueness): Note that $R$ can be written as $R = UV^\top$ iff $A = R \sqrt{A^\top A}$ is a polar decomposition of $A$. If $A$ is not invertible, then $A$ has multiple polar decompositions of the form $A = R\sqrt{A^\top A}$, which means that the minimizer is not unique.
Conversely, if $A$ is invertible, then $A$ has the polar decomposition $A = R_* P$ with $P = \sqrt{A^\top A}$, and we can rewrite our second objective function as $$ \langle A, W \rangle = \operatorname{Tr}([R_* P]^\top W) = \operatorname{Tr}(P R_*^\top W) = \langle R_*, W\rangle_P $$ where we define the bilinear form $\langle X,Y \rangle_P = \operatorname{Tr}(PA^\top B)$. Note that this defines an inner product over $\Bbb R^{3 \times 3}$. Equality holds in the Cauchy-Schwarz inequality if and only if one arguments of the inner product is a positive multiple of the other. Thus, if $\langle R_*, W\rangle_P$ is maximal, then $W$ must be a positive multiple of $R_*$, which (because of the orthogonality constraint) implies that $W = R_*$.
Thus, if $A$ is invertible, then the minimizer is unique.
Note 1: If $UV^\top$ turns out to have determinant $-1$, then a minimizer among the rotations will be $$ U \pmatrix{1&0&0\\0&1&0\\0&0&-1} V^\top. $$ This follows from the discussion of Wahba's problem. I am not sure if uniqueness is guaranteed as it was for the problem over the orthogonal matrices.
Notably, if $\|\operatorname{Re}[\mathcal N_B] [B]_{\text{ref}}^{-1}\|_F < \sqrt{2}$, we can guarantee that $UV^\top$ (the overall orthogonal minimizer) will have determinant $1$, which means that this adjustment would not be necessary.
Note 2: $R\sqrt{A^\top A}$ is a polar decomposition if and only if $R$ maps $\sqrt{A^\top A}x$ to $Ax$ for all $x$ and maps $\ker(A)$ to $\ker(A^\top)$ isometrically.